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Abstract 

This report on the second year of data collection in the Understanding Consequential Assessment 
Systems of Teaching (UCAST) study describes administrators’ learning during the first year that 
Los Angeles Unified School District’s (LAUSD) observations were part of a consequential 
teacher evaluation system for teachers. Drawing on mixed methods, the report provides an 
overview of the Teacher Growth and Development Cycle (TGDC) implementation during the 
2013-2014 school year. It summarizes how much time administrators spent on TGDC-related 
activities and presents their reflections on the strengths and weaknesses of the TGDC 
implementation. The report also documents how administrators used the TGDC process with 
their teachers, how their use of the protocol changed over two school years, and when the 
observation process did and did not work well. We also review data from two in-depth cases that 
reflect wider themes in the sample. 

Key words: principal learning, teacher evaluation, observation, policy, implementation, 
administrators 
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This is the final report of the second year of data collection in the Understanding 
Consequential Assessment Systems of Teaching (UCAST) study, funded by the W. T. Grant 
Foundation. The goal of UCAST is to understand the process by which local administrators in 
the Los Angeles Unified School District (LAUSD) learn to conduct Teacher Growth and 
Development Cycle (TGDC) observations, as part of the district-wide Educator Growth and 
Development Cycle (EGDC) evaluation reform. 

We present findings from our data collection across the 2013-2014 school year, 
beginning with administrators’ overall perceptions of what went well and not so well in their first 
full year of implementation, including how they implemented the TGDC with their teachers and 
how their use of the Teaching and Learning Framework (TLF), an observational tool based on 
Danielson’s Framework for Teaching (Danielson & McGreal, 2000), evolved since the previous 
year. We draw on various sources of data in creating this report, including (a) three bimonthly 
surveys and one end-of-year survey of participating administrators, (b) interviews conducted 
with a group of 38 focus administrators at the beginning and at the end of the school year, (c) 
TGDC think-aloud rating exercises with the focus administrators, (d) field notes from TGDC- 
related professional development at one school site, and (e) interviews with LAUSD central 
office staff members. The specific data drawn on in each section of the report are detailed in 
Appendix A. 

This report focuses on two broad research questions: 

1. How did the 2013-2014 TGDC observations with teachers go? 

2. How did administrators use the TGDC in the 2013-2014 school year? 

In this report, we describe how we investigated these questions, present findings, and 
explore implications based on what we learned. To answer Research Question 1, we begin by 
providing an overview of the TGDC implementation. We then summarize how much time 
administrators are spending on TGDC-related activities and present their reflections on the 
strengths and weaknesses of the TGDC implementation. To answer Research Question 2, we 
investigate how administrators used the TGDC process with their teachers, how their use of the 
protocol changed from Year 1 to Year 2, and when the observation process did and did not work 
well. We also review data from two in-depth cases that reflect themes we have seen across our 
focus observers. 
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Data collection had just completed at the time of the writing of this report, and analyses 
are ongoing. 

Los Angeles Unified School District’s Educator Growth and Development Cycle, 2013-2014 

LAUSD’s new evaluation system, the EGDC, has components for evaluating central 
office staff, school leaders, and teachers. In the academic year 2013-2014, the teacher 
component of the EGDC — called the TGDC — is, for the first time, being used to make 
consequential human capital decisions across the district. Previously, TGDC was implemented in 
a practice year in 2012-2013, following an initial pilot year of the observation system with 
administrator and teacher volunteers in 2011-2012. We reported extensively on our findings 
from the 2012-2013 academic year in last year’s final report (Bell, Jones, Lewis, et al., 2013). 

The observation instrument, called the TLF, is a modified version of Danielson’s Framework for 
Teaching (Danielson & McGreal, 2000). Danielson’s original instrument is said to be the most 
widely used observation protocol in the country. The new instrument, the TLF, has been aligned 
to the California teaching standards and contains modifications to reflect the values of LAUSD 
stakeholders. 

The TGDC is a significant departure from past evaluation practices in LAUSD. The 
system requires administrators be trained and certified in ways that are unprecedented in Los 
Angeles. The original training and certification mechanisms are beyond the scope of this report; 
however, it is important to review briefly the training that occurred. 

All administrators in the district are required to be certified on the TLF. Certification 
includes participating in a weeklong training session and passing a certification test. The 
certification test requires the administrator to review a lesson plan, watch the video-recorded 
lesson, and review the administrator’s postobservation conference with the teacher. The 
administrator does this by following a process of taking notes on the video-recorded lesson, 
aligning 1 observed evidence of teaching quality with the TLF rubric, and rating the lesson. 
Certification has four components: the accuracy of scores, the degree to which the 
administrators’ notes are free of bias, the extent to which observation notes are aligned with the 
TLF, and the degree to which the notes are representative of the lesson. Administrators are rated 
on each component of the certification test and must receive an overall score of preliminarily 
certified to conduct observations independently. Administrators who do not meet this standard 
must study and retake the test. 
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These quality-control mechanisms have been put in place to support the development of a 
common set of procedures and standards that can help principals and teachers work together to 
improve teaching and learning in their buildings. The processes also serve to improve the 
validity of TGDC ratings so that they can be used in human capital management decisions. For a 
detailed analysis of LAUSD observer certification rates and the implications for evaluation, 
please see last year’s report (Bell, Jones, Lewis, et al., 2013). 

Implementation Summary 

During the 2013-2014 school year, all school administrators who were trained and 
certified on the TGDC were asked to complete observation cycles with multiple teachers in their 
schools. To understand the implementation of TGDC during 2013-2014, we collected extensive 
data from 38 purposefully selected focus administrators. These administrators participated in 
surveys, interviews, and think-aloud exercises. For administrators who had been in our sample 
for 2 years ( n = 23), we also drew on similar data collected during 2012-2013 to track how 
administrators learned to implement TGDC over time. 

Description of Focus Administrators 

Focus administrators were selected to mirror the population of administrators in LAUSD. 
In defining our sample, we attended to administrator role (e.g., principal, assistant principal), 
instructional level (e.g., elementary school), time of TGDC training, and location or educational 
service center (LAUSD is divided into five local educational service centers). Of our 38 
participants, over 50% were principals, and another 25% served as assistant principals. Four 
administrators were instructional directors, and three served as instructional coaches or 
specialists. About 40% of the focus administrators worked in elementary schools, 20% worked in 
middle schools, and 33% held positions in high schools. The administrators had an average of 
4.4 years in their current professional roles, including time in LAUSD and any other district. 
Slightly more than half of the focus administrators were female. Twenty-three administrators 
also participated in our study during 2012-2013. 

Of focus observers’ schools for which data were available (n = 37), an average of 82% of 
students received free or reduced-price lunch, and 5% of schools made adequate yearly progress 
during the 2012-2013 school year. As of the 2013-2014 school year, the student populations 
served by our focus observers’ schools were, on average, 80% Hispanic/Latino, 7% White (non- 


RM-15-12 


3 



C. Bell et al. 


Understanding Consequential Assessment Systems of Teaching 


Hispanic), 5% Black/African American, 4% Asian, 2% Filipino, and 0.4% other (California 
Department of Education, 2013). Additionally, 8% of schools participated in the LAUSD Pilot 
Schools Program (Los Angeles Unified School District, 2013). 

How Administrators Selected Teachers 

To reduce burden on administrators, they were not expected to observe all teachers in 
their schools during 2013-2014. Instead, focus administrators were allowed to select a subset of 
teachers to evaluate formally. In our initial interviews, we asked administrators how many 
teachers they were formally evaluating and how they chose which teachers to evaluate in 2014- 
2015. Focus administrators reported an average caseload of 5 teachers (M = 5.0, SI) = 2.40) for 
whom they were obligated to conduct TGDC observations during the 2013-2014 school year; no 
focus administrators worked with more than 1 1 teachers. 

Administrators selected teachers to participate in a variety of ways. District policy 
required teachers on probationary status or in a contractual observation year to participate in the 
TGDC cycle. In addition to teachers for whom participation was mandatory, focus administrators 
selected (a) teachers perceived to be struggling, (b) teachers perceived to be able to benefit most 
from the TGDC process, (c) teachers who were new to the school but not necessarily new to the 
profession, (d) teachers who volunteered, and (e) teachers who were selected in a manner 
administrators characterized as random. 

Focus administrators paid attention to how teacher observation responsibilities were 
distributed between themselves and fellow administrative staff. They tried to avoid a situation in 
which any one individual would observe too many or too few teachers. In cases of multiple 
administrators in one school building, which occurred largely at the middle and high school 
levels, formal and informal TGDC observation work was sometimes coordinated among them. 
Our data show two primary ways in which this happened: either a caseload of teachers was 
(roughly) equally divided across administrators (principals and assistant principals) or principals 
took on most observations but relied on assistant principals and other supporting administrators 
to conduct observations in the event that the observation schedule became too burdensome. 

How Much Time Was Devoted to Observations? 

As noted in the UCAST Year 1 report (Bell et al., 2013), focus administrators sometimes 
raised concerns about the time burden of the TGDC process once they were working with more 
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than one teacher. With this in mind, we asked administrators several questions during and after 
the school year to estimate the amount of time they devoted to TGDC. On average, focus 
administrators spent approximately 7 hours per week on all TGDC-related activities (see Table 
1) and approximately 12 hours (M = 1 1.94, SD = 7.42) conducting a full TGDC observation 
cycle, from pre- to postobservation conference, for one teacher. This represented a substantial 
proportion of the overall amount of time that administrators reported working per week — 44 
hours on average. On a weekly basis, this means that focus administrators spent about 16% of 
their time on TGDC-related activities (7 hours/44 hours). Over the course of the school year (36 
weeks), an average administrator devoted 252 hours to TGDC-related activities — or 5.7 weeks 
(252 hours/44 hours per week). 2 Because the average caseload in our sample was five teachers, 
we estimate that focus administrators in our sample spent, on average, 120 hours (5 teachers x 12 
hours x 2 observations) total on the formal observation component of the TGDC process. It is 
important to note that there was variation around this average (Table 2). Additional TGDC 
activities included informal observations, setting up meeting times, and professional 
development associated with the cycles. It is important to note that focus administrators varied 
dramatically in how long they spent on the TGDC observations, all tasks related to teacher 
evaluation, and their jobs overall. 

Instructional Directors’ Experiences With the Teacher Growth and Development Cycle 

In addition to the principals and assistant principals included in our sample of focus 
administrators, the sample also included a small number ( n = 4) of instructional directors. Each 
instructional director was assigned to one principal who was participating in the School Leader 
Growth and Development Cycle (SLGDC). The SLGDC included a series of reflection activities, 
formal and informal visits, conference opportunities, and professional goal setting aimed at 
identifying strengths and opportunities for improving school leader practice. Instructional 
directors’ understanding of the TGDC process was therefore informed by their practice with the 
SLGDC process. 
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Table 1. On Average, How Many Hours per Week Did You Spend on the Following Tasks? 


Task 

N 

M 

SD 

Min 

Max 

All tasks related to formal and informal Teacher Growth and 
Development Cycle observations in 2013-2014 

31 

7.48 

17.45 

1 

100 

All tasks related to teacher evaluation 

31 

6.84 

4.80 

1 

20 

All tasks related to your job 

31 

44.48 

19.65 

1 

70 

Table 2. What’s the Shortest and Longest Amount of Time You Spent on One Formal 


Observation Cycle for a Teacher (Hours)? 






Time 

N 

M 

SD 

Min 

Max 

Shortest 

19 

7.68 

4.66 

2 

20 

Longest 

22 

13.86 

8.31 

3 

30 


In addition to the described SLGDC activities, instructional directors engaged in TGDC 
activities in a number of ways: (a) taking on a caseload of teachers to alleviate the burden on the 
principal they supported; (b) providing general advice and support within their assigned schools; 
(c) actively coaching their principals in conducting the TGDC process, providing concrete and 
actionable feedback when necessary; and (d) indirectly monitoring the TGDC process at their 
assigned school site(s). 

Strengths and Weaknesses of the Teacher Growth and Development Cycle 
Summary of Findings 

• Focus administrators expressed a positive view of the overall TGDC process and 
considered it to be a useful system to support educators. 

• The most frequently discussed strength of TGDC was the TLF. 

• Some focus administrators reported that they noticed improvement in technology as 
well as in teachers’ attitudes. 

• More than half of the focus administrators reported that what did not go well was the 
huge demand on their time; most identified that as the primary weakness of TGDC. 

• Technical difficulties were still one of the most frequently mentioned weaknesses of 
TGDC and a reason for things not going well this year. Accordingly, focus 
administrators frequently suggested teacher training and additional technical and 
human resources support. 

• Focus administrators wanted opportunities to discuss TGDC with colleagues. 
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Background 

Focus administrators were asked during initial interviews to share their thoughts on what 
went well and what didn’t go well during the TGDC rollout in 2013-2014. Later, at the end-of- 
year interview, they further commented on the strengths and weaknesses of the TGDC. In this 
section, we draw primarily on these two data sources to describe the views and experiences of 
focus administrators. We also support some of the findings using results from our bimonthly and 
end-of-year surveys. 

What Went Well: Strengths of the Teacher Growth and Development Cycle 

Observers identified the objectivity, fairness , and comprehensiveness of the TGDC as its 
strengths. For example, Lana, an elementary school principal, expressed a positive view toward 
the overall TGDC process because “it’s honest; it’s objective; [and] it follows a pattern of 
integrity.” Ron, also an elementary school principal, said that sometimes parents asked him 
whether a particular teacher is good, and he said, “If you asked me 5 years ago, I would give you 
a subjective opinion.” He felt his evaluation of teachers had become more objective due to the 
TLF. Harvey, a high school assistant principal, believed that the TGDC “is more fair and it looks 
at the overall aspect” of educators’ work; he also thought that the “standardization of what to 
look for” was helpful. Assistant Principal Vicky felt that TGDC was good in the sense that “it’s a 
really structured protocol.” 

Some focus administrators also appreciated that TGDC provided opportunities for people 
to get targeted feedback and improve their practice. Elementary school principal Donna said, “I 
enjoy it since we get to spend more time with teachers.” She also thought that it “does help 
instruction.” Edger, an assistant principal at an elementary school, believed that the strength of 
TGDC is that “it does provide teachers with targeted feedback. In the past, some administrators 
were not doing that part.” Ella, an instructional director, was glad that “we are taking giant steps 
rather than baby steps, in the growth of the principals and teachers. It’s worth it.” 

Quite a few administrators emphasized that teacher reflection on instructional practices is 
a strong and valuable piece of the evaluation process. Charles, an elementary principal, believed 
that the strength of the TGDC is that “teachers going through the process have a chance to reflect 
on lessons and have a chance to be coached by a principal. Some teachers really like cognitive 
coaching.” When reflecting on what went well with TGDC this year, Vincent, a high school 
assistant principal, thought that he agonized over the ratings and probably spent too much time 
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doing them, “but the end result is that the teacher really reflects on his or her practice. I honestly 
wish we had something like this when I was in the classroom. Just the self-assessment alone is 
worth the process.” 

During the semistructured interviews, one of the most often identified strengths of the 
TGDC was the TLF. Focus administrators truly embraced the framework and regarded it as a 
powerful tool to guide the evaluation process and initiate conversations about improving 
instruction. This finding has been consistent across the 2012-2013 and 2013-2014 school years. 
In our 2013-2014 end-of-year survey, at least 70% of the focus administrators agreed with the 
following: 

• The TLF covers important domains of teaching. 

• The TLF is a valuable tool for talking to teachers about instruction. 

• The view of instruction underlying the TLF is similar to my view of instruction. 

Lydia, a principal, explained, 

Obviously, being evaluated raises everyone’s level of tension and pressure, but I think 

having a rubric that you can really speak to is really helpful; makes it evidence-based. 

Having the rubric guide you to where you want to be is very powerful. 

Charles told us that the strength of the TGDC is having the TLF “where the teachers are 
provided a list of the elements so they know what quality teaching looks like.” Andy, an assistant 
principal at an elementary school, indicated that his teachers respected the framework: “They 
don’t disagree. I haven’t had anyone saying anything negative about the TLF; no one denies it’s 
useful and effective.” Portia, also an assistant principal, liked the rubric because “it takes all the 
blame out of it and it depersonalizes the process.” 

A few focus administrators reported that what went well this year was their teachers’ 
improved understanding of the purpose of the TGDC. The administrators accomplished this 
through better communication and sharing other teachers’ experiences, which partially reduced 
teachers’ anxiety and fear about the process. Cathy, an assistant principal at a middle school who 
worked with a volunteer teacher during the practice year (2012-2013), told us that “the good part 
is that I’ve done some of it so the word is out there, that it’s not a bad thing.” Portia felt that 
she’s “better able to communicate to the teachers what their strengths and needs are, and in 
return the teachers are more open.” Ron said that none of his teachers resisted meeting with him: 
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“It has to do with my approach with the teachers.” Both he and his coach told the teachers that 
“the goal of this process is to assist you as a learner.” Gary, a principal at an elementary school, 
explained that what worked well this year was having conversations with teachers in advance 
“not to be scared of it — it’s coming, but it’s also a way to improve [your] practice.” Although 
many teachers were still nervous and had doubts about the evaluation process, focus 
administrators’ experiences suggested that strong communication helped to ease teachers’ 
anxiety. 

Compared to the practice year, a few administrators told us that they were more 
comfortable and familiar with the TLF and the TGDC process during the 2013-2014 school 
year. Mary, a middle school principal, said, “I think I debrief well with my teachers. My 
experience as a mentor teacher helped. A year ago, I was looking at the elements when talking 
with my teachers. I feel more comfortable now.” Timothy, also a middle school principal, felt 
that tagging (i.e., aligning evidence collected during observations to TLF elements) and 
debriefing went well this year: “I’m really good at matching up the evidence to the standards. . . . 
I’m really good at showing [teachers] the evidence and not making them feel inferior about it.” 
Lana expressed that she felt “more secure; it’s not so bad.” Gary told us that he felt that the 
practice year was new learning for administrators as well as for the teachers. Teachers in his 
building began to see how the TGDC is connected to what they do “so that it’s integrated and not 
isolated.” 

Some focus administrators thought that what went well in 2013-2014 were technology 
improvements, particularly to the online scoring platform. My Professional Growth System 
(MyPGS). They acknowledged that the platform became more user-friendly and ran much more 
smoothly than during early phases of the TGDC implementation. Sally, a high school principal, 
said, “There have been a lot of improvements. There are a number of things we’ve brought up to 
a TLC [teaching and learning coordinator] that could still be improved. But it’s vastly improved 
from the beginning.” 

Several focus administrators commented that the lead teachers at the school level and 
TLCs at the district level were helpful during the 2013-2014 school year rollout. Andy told us, 

We do have coordinators at the district level. I know some principals have asked them to 
come out to help. That’s helpful. The teacher can go to a [lead] teacher for help. She’s 
universally trusted by the staff. It’s a very good strategy. 
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Ron also thought that the “lead teacher was very helpful to the teachers being evaluated. 
She met with them periodically and met with them individually.” Elizabeth, an elementary 
school principal, told us that sometimes she called her TLC on the weekends and “she always 
answers me and gives me support.” 

What Didn’t Go Well: Weaknesses of the Teacher Growth and Development Cycle 

As in the first practice year, focus administrators’ most common concern about the 
TGDC process was still the time burden. More than half of the administrators said the process 
was too time consuming. Across the board, focus administrators thought that the amount of time 
it took to complete the TGDC process was excessive. Their concerns with the time burden 
included being able to keep up with the amount of work required to complete the observation 
cycles faithfully, especially on top of the operational demands of running their schools. Portia 
expressed the stress she felt: 

[The process] is extremely time consuming ... for that reason it’s undoable. In order to 
get this done, we already work a 10-hour day, and on top of that we have to do this at 
home or on the weekends to get it done. I have a staff of 38 teachers. If I were to evaluate 
the teachers every other year as recommended, that’s insane. I cannot meet the deadlines; 
it’s really unrealistic. 

Second, focus administrators felt they didn’t have enough time to conduct the TGDC and 
focus on instruction and coaching as they would like to, leading to uneven implementation. One 
instructional director told us that he “continued to hear complaints about how much it takes to do 
the work. I see it not being implemented consistently. You are supposed to do a preconference, 
[but] I have principals who were not doing them.” Sara, an elementary school principal, said, 
“One person can’t do this and do operations. ... You can’t be a manager and be an instructional 
leader. . . . We could do it so much better if we had the time. ... You try to cut comers and 
things like that.” Portia told us that she would like to improve her coaching, but that 

it takes practice and consistency, and time, lots of time to coach. And being an 
administrator it’s very difficult because you have a lot going on, there’s just so many 
things happening during the school day and they’re all operational. . . . Since this is 
unrealistic because it’s time consuming, principals are exercising their right to give 
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waivers. If I have 14 [teachers] next year, I will chose 4 or 5 at the most and give waivers 
to the others. 

Third, administrators felt that the TGDC took their time away from nonparticipating 
teachers who still needed administrators’ attention and support. Andy told us, 

If I evaluate four people, that means that 19 won’t get evaluated this year. In a way, it’s a 
good thing since I can keep focusing on those four. But then other people get passed 
over — quite a few teachers are not in this process at all for three or four years, or even 
longer. Everyone needs to improve. 

Mary was also concerned that she did not have time to help the teachers who are not 
participating in the TGDC this year. 

Fourth, a few focus administrators reported that the TGDC was a challenge for their daily 
time management. In their survey responses, more than half (53%) of focus administrators said 
they needed support with their time management. Mary explained, 

With everything else you have to do, how do you do this well? If this was all I have to do, 
I would be able to do it well. But with everything else I do with my professional life, my 
personal life suffers. 

Vicky said that it was “difficult to find uninterrupted time to do [the TGDC]. Even 
though I blocked time on my calendar, things come up.” Bridget, a high school principal, told us, 
“I would like to get better at planning my time and sticking to my plan. I would also like to 
spend more time talking about the instruction versus explaining the [TGDC] process.” 

Although some administrators acknowledged that they had noticed improvements to the 
online platform, technical difficulties were still the second most frequently mentioned weakness 
of the TGDC. Donna commented that “the web system is not user-friendly, too many things you 
have to do (highlight and tag), go back and forth throughout the system. [It] takes too much time. 
[The] system needs to be very simple.” Scott, an instructional director, expressed a similar 
opinion: “I think the things going in [the system] are important but they could be simplified.” 
Lydia told us that “the technical difficulties with the platform have been frustrating for some 
because not everyone has facility with technology. For some of [my teachers] it’s been very 
uncomfortable.” Timothy was disappointed with the capacity of the web system “when everyone 
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[was] trying to finish the first cycle [in December,] it was frustrating because it seemed like the 
system was slowed down.” 

A handful of focus administrators expressed frustration about a lack of communication 
about upcoming plans for the evaluation and use of the TGDC data at the end of the year. Sally 
thought one weakness of the 2013-2014 implementation year was “probably the communication 
of what’s the timeline for establishing benchmarks.” She continued, 

We need to know what we’re doing with this information by the end of the year. How do 
we determine whether teachers meet a certain standard? Without knowing where we’re 
going, it’s very hard. I admire the pushing forward but we still need to know where we’re 
going. 

One administrator said he didn’t know whether “LAUSD is going to be married to this 
platform. ... I don’t even know what direction we are going.” Lydia explained how this sense of 
uncertainty caused problems: 

There’s this sense of at the end: Who decides? The idea of scoring hasn’t been negotiated 
with the union yet so people don’t know what the expectation is, and the fact that that 
hasn’t been worked out, it makes people nervous and anxious. 

Several administrators felt the documentation and the end-of-year evaluation did not fully 
align with the TGDC work they did during the school year. Mary commented that, at the very 
end of the school year, when they had to rate teachers, “the column Highly Effective was gone. 
And if a teacher gets a Highly Effective, the final results [are] not show[n] to the teacher.” 
Timothy said, 

[The] TGDC platform is based on Highly Effective, Effective, Developing and 
Ineffective. At the end of the year, when I put my ratings in the system, the rating choices 
were just “meet/doesn’t meet”; there’s no middle ground. ... So I just had a question 
about how it will be in the future. 

Gary identified the final evaluation rating as the weakness of the TGDC, because “we can 
do the same thing we’ve been doing to get the end evaluation done.” Darlene echoed Gary’s 
concerns, pointing out that spending a lot of time and effort to evaluate a teacher on the 4-point 
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scale only to have to rate them as meet/doesn’t meet was frustrating and seemed contrary to the 
district’s messages about using the TLF. 

Focus administrators also talked about how teachers were still somewhat resistant to the 
evaluation process. This was partly due to the lack of communication about the purpose of the 
TGDC. Bridget said, “Most people don’t understand it. . . . We’re so bad at implementing things 
and transitioning into new things. No one understands why we are doing things.” Cathy 
connected this to prior evaluation experiences, explaining, 

The bad part is that we started this year off with many people having a bad attitude about 

[TGDC] . They were basing [it] on the STULL; they were using STULL as the source of a 

negative attitude. They thought it’s just going to be another thing to do. 

Similarly, Charles felt that “some teachers don’t like to be coached; they want to be left 
alone.” Martin, an assistant principal at a magnet school, thought that “it’s great that the new 
evaluation system is aligned to the new curriculum. But because it’s asking teachers to change so 
much in many different ways at a time, it’s going to cause a lot of anxiety and fear.” 

Several administrators said there were too many changes to the process and the rubric, 
which made it hard for them to keep up. Vicky recalled, “Everything was in draft form when I 
had to do this [during the practice year]. There have been some changes.” Sara expressed her 
frustration: “Why are we adding new things before refining just one thing? Stick with what we 
got and do it well. You can’t do it [all] in a year.” 

Some administrators still felt that parts of the process were difficult. For example, some 
found scripting to be challenging and others struggled with tagging. A few people felt that the 
rubric was still up to one’s interpretation and more guidelines were needed. Cathy cited tagging 
as a weakness of the TGDC “because it is still up to your interpretation. For example, even 
though we went through the training, you tag it, but it’s up to your interpretation. It’s very open.” 
Donna echoed this idea: “TGDC has guidelines that are far less than what is actually needed. If 
you really want to change, you really need to spend the time to create the detailed guidelines.” 

When asking 38 leaders how well the rollout of a new staff management system is going, 
it is not uncommon to get wide-ranging and contradictory responses. That said, administrators’ 
consistent concerns about the burden of the system and mixed data on teachers’ comfort with the 
TGDC suggest that these areas merit further discussion. 
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What Supports Are Needed 

Focus administrators were asked what supports or resources would be helpful in making 
the TGDC more effective and their work easier. Suggestions ranged from supports for teachers 
to supports for the administrators themselves. Quite a few felt that it was very important that all 
teachers get proper training that addresses their needs and concerns and helps them understand 
the purpose of the TGDC. Gary said, 

I want the teachers to all have the training; I want them all to embrace it and it doesn’t 
need to always come from me. I think I’ve said it enough and the district needs to do that. 
And those teachers who don’t need to be STULLed should be required to go through the 
process so that they understand the process. A lot of people are scared, not wanting to do 
the process, especially those who have been teaching for 20 plus years who feel they 
don’t need it. They need to go through the process and understand it’s a help[ful] tool, 
and it will help them become experts in certain areas and see . . . how the TLF will help 
them be the best teacher ever. 

Mary believed that teacher training needed to be mandatory because “they need to learn 
what they have to do. They’ll feel more comfortable.” Ron also thought more training would be 
helpful: “I don’t think teachers understand the rubrics; they could be more familiar through 
discussion, demonstration lessons, [and] rating lessons themselves. That’s what we don’t have.” 

Several administrators expressed the desire to have the opportunity to discuss the TGDC 
with colleagues. This need was reinforced by the results of our bimonthly focus administrator 
surveys, in which half (50%) of focus administrators said they needed opportunities to discuss 
and collaborate with other administrators on the TGDC implementation. Sara acknowledged that 
she had lots of support at the district level, yet 

I need support at the school level. We used to have instructional coaches for me to 
brainstorm with. Having a colleague to talk with, someone who is seeing the same 
teacher and can agree with you, or could say I saw something else when I was in the 
room [would improve the process] . 

Portia was also concerned about having to learn and grow in this new work alone: 
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When I do this, I don’t have anybody guiding me. I’m alone and I’m trying to do the best 
that I can, and I never know if it’s accurate or if I’m putting [observation data] in the 
right place. I don’t have a collaborator to make me grow as a better evaluator. 

It is worth noting that Portia felt fairly confident using the TLF and guiding teachers in 
their development during the practice year (2012-2013). It may be common for administrators to 
underestimate their learning needs when first using the TLF and then identify areas of weakness 
or confusion as familiarity with the tool increases. Future analysis of Year 1 and Year 2 
interviews with administrators will investigate the prevalence of this phenomenon. 

Andy shared positive experiences with having a co-observer during the TGDC practice 
year: “That was really powerful, both for me and the teacher. [It was a] great learning 
experience, with someone else.” He liked getting feedback and “also having conversations about 
what we both saw, what we liked and what we didn’t like.” Lydia also thought collaboration 
among schools to share best practices was very helpful: 

Principals that have done it before shared their best practices. We’re collaborating with 
five other schools and we have weekly meetings. . . . Most of those meetings were about 
the TGDC, just all the different aspects, that was helpful too. It’s very timely and very 
much organic because it’s based on what we’re going through in real time. 

Focus administrators also told us that they would like to see further technical 
improvements to the system, including additional technical support and training on specific 
aspects of the process. In our bimonthly focus administrator surveys, almost one-fourth (24%) of 
administrators indicated that they needed technical support using the online platform. For 
example, Lana thought that the platform needed to be modified to make the evaluation data more 
accessible to teachers so that it can then promote deeper learning: 

After the administrator has done lesson design review, teachers should see those 
comments connected to where I put the notes in the platform. Then there’s a one-to-one 
correspondence. It’s real learning. I’m giving my time to write these, thinking they are 
guiding questions for the teacher, and they can’t find them or how they’re connected. 

A few administrators hoped technology support or other resources would be available to 
streamline the process. Martin told us that 
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administrators need support in the process because it’s so lengthy. Whether it’s less 
teachers or some sort of technological advantage to record the teachers’ classes and then 
have everything scripted out. Something like that so we can have the teachers’ words 
written down and we can focus on the students. 

Ron suggested that technological support from additional staff would be “phenomenal,” 
specifically a staff member to help input the observation data: 

I can do the observations, but finding the time after work [to input the data] is hard. 
Literally, I come in on Saturdays and spend 5 hours, because of the attention to detail. 
[For] one teacher I have 1 1 pages to transcribe. My wife helped me. It takes longer this 
year; I have more teachers . . . and it is impossible to do it right after the observation 
because of reflection time teachers need. 

Although administrators’ ideas about the particular aspects of technology varied, the 
interviews suggested that there is still clarity and efficiency to be gained from further trainings. 

A few administrators felt that they needed support around coaching conversations specifically. In 
our bimonthly focus administrator surveys, more than one-third of observers wanted help having 
coaching conversations with teachers. Sally told us, 

I think that principals would benefit from more examples from our framework and one of 
our principals having a conversation with one of our teachers. They’ve shown us clips but 
they’re from other contexts — not even using the framework. It’s not the same kind of 
conversation. It’s asking people to transfer that across contexts. It would be helpful for 
people to see how you talk about specific evidence. I don’t know how you could expect 
people to change how they do things without some sort of modeling. 

Maria, a middle school principal, similarly felt that she was not good at postobservation 
reflection: “What I get stuck on is having and finding better ways of asking questions that cause 
the teachers to stop and think.” Maria’s concern was consistent with the literature on cognitive 
coaching and instructional leadership. Having productive instructional conversations about how 
to improve teaching practices is and has been a challenge across time and place. 

Although, in general, administrators with whom we spoke were quite positive about the 
TLF, some worried that the current rubric didn’t apply to certain groups of teachers, such as 
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special needs or art teachers. They thought additional resources and supports for these teachers 
should be developed. 

Finally, three focus administrators mentioned that some procedures or processes needed 
to be put in place to ensure the accuracy and reliability of their ratings. Sally told us, “One other 
thing that’s missing is that we haven’t done any calibration. We haven’t come back to 
certification. I can see myself drifting away from the way we’re supposed to do things.” Charles 
said, “I have to go and get rated to make sure that I’m tagging correctly; that’s rater reliability — 
what I see and what another observer sees.” Vincent echoed this point: “We all need calibration, 
over time.” These administrators’ concerns about accuracy may reflect differences in how 
administrators used the TLF, as later sections of this report suggest. It may be that 
administrators’ awareness of these differences made them want to be better calibrated with one 
another. 

Administrators’ Use of the Teacher Growth and Development Cycle Process With Teachers 
Summary of Findings 

• Focus administrators used information from other interactions with the teacher (e.g., 
walk-throughs) to inform aspects of the TGDC cycle. 

• Many administrators tailored the logistics to the teacher (e.g., length of pre- and 
postobservation conferences, location, number of cycles). 

• All administrators scripted lessons. 

• Administrators varied in their willingness to depart from the prescribed TGDC 
process. 

• Focus administrators used the TGDC for both evaluation and improvement purposes 
but overwhelmingly focused on supporting instructional improvement. 

• Administrators’ goals for teachers varied between individuals and over time. Goals 
were sensitive to the specific contexts of the people and schools involved. 

• Administrators developed many strategies to improve TGDC implementation. The 
strategies addressed five foundational issues — efficiency, scheduling, transparency, 
trust, and improvement. 
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Background: The Teacher Growth and Development Cycle as a Tool for Administrators 

TGDC documentation and interviews with central office staff suggest that LAUSD has 
consistently viewed the TGDC as a tool for administrators and teachers to develop a shared 
understanding of teaching and how to improve teaching. In this section, we consider the ways in 
which focus administrators used the tool with teachers who had both positive and negative 
experiences with the TGDC in 2013-2014. We focus in particular on the ways in which 
administrators’ goals, knowledge, prior experiences, orientations toward teacher learning and 
supervision, school contexts, and issues specific to the teacher shaped administrators’ use of the 
TGDC. 

Where, When, and How Teacher Growth and Development Cycles Happened 

In the Implementation Summary section, we described some general trends in how 
frequently TGDC cycles happened. Here we describe more qualitatively the details of how 
administrators carried out the cycles. First, and perhaps most important, the cycles recorded on 
the online platform are not the only source of observational information used to assess and 
support teachers. Some focus administrators made extensive use of additional information 
obtained in walk-throughs or other informal interactions around a teacher’s practice. 
Administrators tended not to document these observations in a formal way, so it is difficult to 
determine how these supplemental observations occurred and how prevalent they were across the 
district. 

Second, administrators varied as to where they held pre- and postobservation 
conferences. Administrators in one region reported that they were encouraged to conduct all of 
the pre- and postobservation conferences in teachers’ classrooms. However, this was not the 
norm among the focus administrators. They most often reported having conferences in their 
offices; however, there were administrators who reported a deliberate attempt to meet with 
teachers in their classrooms. One observer, a high school principal, mentioned that he thought it 
was better to do the conferences in the teacher’s classroom, but sometimes, when pressed for 
time, it was easier to have the teacher come to his office. Administrators who reported a regular 
practice of going to teachers’ classrooms often did so because they wanted to send the signal that 
the TGDC was about the teacher — collegial and focused on improvement, not evaluation. 

The time administrators took to conduct the pre- and postobservation conferences varied 
both over time and across individual administrators. Some tried (for fairness) to make all the 
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conferences the same length (i.e., 15-45 minutes). Others tried to tailor their conversations to the 
specific needs of the teachers, spending more time with teachers who were struggling or who 
wanted to discuss the lesson in detail. The length of time used for a pre- or postobservation 
conference seemed to be based on the administrator’s workload (i.e., if the administrator could 
spare the time for a longer conference), the level of detail at which the administrator planned to 
engage the teacher, how quickly the teacher and administrator were able to get through the 
primary issues each needed to discuss, and the chattiness of the individuals involved. 

Finally, one area of practice that was widely shared across administrators was scripting. 
Administrators discussed the practice of scripting (both its strengths and weaknesses) and 
generally reported that they scripted as best they could. It is unclear how prevalent scripting was 
from these data, but in general, administrators valued the scripts they created for the observed 
lessons. It is important to note that scripting was emphasized as a requirement of the TGDC 
process and was included on the certification test. 

Looking more holistically at the ways in which administrators carried out the cycles, 
some were very committed to the TGDC process as it was specified in the training. They did not 
want to skip steps; they used the various tools provided on the platform and during training 
sessions. Other administrators made modifications, such as doing fewer observations (both 
informal and formal) of teachers they felt were teaching acceptably and completing the 
recommended number of observations for teachers who needed more support. Still others 
developed streamlining strategies that modified the process (e.g., having a preobservation 
conference with two teachers at once). We describe these approaches at the end of this section. It 
is worth noting that we do not know the prevalence of these approaches. In the section Lessons 
Learned: A Case Study of Two Principals, more detail is offered regarding how administrators 
tailored their use of the TGDC process. 

Goals for Teachers 

Using an observation tool implies that the user has a goal in mind. If we take seriously 
the idea that the TGDC is a tool for achieving administrators’ goals, it is important to consider 
what goals administrators had as they began working with teachers. In our interviews with 
focus administrators, we asked them to specify the goals they had going into the TGDC for two 
of their teachers: one teacher the administrator thought would report a positive experience with 
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the TGDC and one the administrator thought would report a more negative experience with the 
TGDC. 


The data suggest that administrators attended to both of the official TGDC goals — the 
improvement of teaching and the evaluation of teachers. Philosophically, many administrators 
agreed that both goals are important. However, the first goal was dominant in administrators’ 
stated goals for teachers. Counting each teacher for whom data were coded separately, roughly 
86% of the goals were focused on improvement, whereas 14% were focused on more evaluative 
goals. Table 3 shows the range of goals that were reported and an example of each type of goal 
in the words of the administrator. 


Table 3. Examples of Administrators’ Goals for Individual Teachers’ Teacher Growth and 
Development Cycles 

Goal Example(s) 


Improvement 

1. To reach more learners and learners who are not 
typical 


2. To support teacher learning on other school-wide 
instructional goals (e.g.. Common Core State 
Standards, cognitive demand, hearing more student 
thinking, providing real-world contexts for math 
problems, implementing problem-based learning) 

3. To work with the teacher on instructional issues 
specific to the teacher’s needs (improving classroom 
management, improving cognitive demand, having 
more productive discussions, slowing down for 
student understanding, looking carefully at who is 
spending time talking, differentiating for English 
language learners, etc.) 


4. To provide feedback to the teacher 


My goal was for her to be able to engage struggling 
students and to differentiate to teaching because 
that’s very important especially when kids have 
special needs. (Portia) 

Real context for math, problem-based learning; 
same goals as for the school. (Anthony) 

Get kids talking; same as school goals. (Bianca) 


Change some of her teaching habits, to incorporate 
various strategies, use manipulatives, do more 
planning. (Andy) 

Looking at the ways she is able to guide discussions 
in math, be able to help students use their 
knowledge and understanding to solve problems. 
Pare down the discussions to be more productive. 
We also focused on the problem of the day; this was 
different than her grade -level goal, which was 
writing. (Lydia) 

Help him make the pedagogical transition to middle 
school from high school teaching. (Timothy) 
Classroom management and questioning. (Lana) 

1 didn’t know what her strengths and weaknesses 
were as a teacher. My goal was to go in and support 
her in her identified area of need. (Charles) 

To help the teacher meet her goals, be a resource, 
provide the feedback. (Portia) 
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Goal 

Example(s) 

5. To get to know a beginning teacher, develop trust, 
and incorporate the teacher into the school community 

Developing a professional relationship. Be 
welcoming and supportive. (Harvey) 

To learn more about Eryn’s teaching, to get to know 
Eryn. (Elise) 

Evaluation 


1. To document a teacher’s practice and response to 
administrator evaluation (in conjunction with the 
conference memo process) 

To get him out of the teaching profession before 
any other children were hurt. (Charles) 

2. To monitor and address foundational issues of 
teaching (attendance, grading practices, honesty, 
treating colleagues professionally) 

We have to work on the timeliness of her 
paperwork. She has to do logs on her work with the 
students. We have to get to a system where she’s 
doing them weekly, keep her on top of it. (Rob) 

I spoke to her about attendance, how she 
communicated with others, sarcasm, relationships, 
impatience about learning. (Bianca) 

3. To get through the requirement of the TGDC 

With her, I just wanted to get it over with. (Danya) 

No specific goals other than let’s get through this 
together. (Harvey) 


Focus administrators did not necessarily see the goals of improvement and evaluation as 
separate. They noted that, as administrators responsible for everything happening in a school, 
they had both to help teachers learn and to evaluate them. Charles, an elementary school 
principal, explained, 

My role is to both support the teachers and give them the materials and resources and 
time they need to do their job, but I also need to evaluate them. When the relationship 
isn’t there and the person sitting on the other side of the desk feels like they’re being 
singled out ... it is hard to work with someone who doesn’t want to be better. 

Charles and other administrators noted that the ability to work on improvement goals was 
contingent on their relationship with the teacher as well as on the teachers’ desire to improve. 

We return to this point later in our analyses of administrators’ explanations of success and failure 
(see the section When Did the Process Work Well? When Did It Go Poorly?). 

Despite evidence that administrators supported both improvement and evaluation goals, 
their goals sometimes differed among teachers in the same building. For example, for one 
teacher, the administrator might focus on improving classroom management and questioning 
techniques, but for another teacher, he or she might focus on improving the teacher’s attendance 


RM-15-12 


21 



C. Bell et al. 


Understanding Consequential Assessment Systems of Teaching 


and student grading practices. Administrators’ efforts to tailor their TGDC goals to the specific 
needs of their teachers aligned with the district’s view that the TGDC can and should be 
individualized to the needs of the teacher. As Table 3 shows, there was a wide range of 
articulated goals. The variation in administrators’ goals appeared to depend, in part, on what an 
administrator knew about the teacher, what specifically the teacher brought to the interactions, 
the administrator’s prior experiences with the teacher, the school’s goals, the history of the 
administrator in the school, and his or her professional values. 

A systematic treatment of each of the factors that shaped administrators’ goals is beyond 
the scope of this report; however, a representative example from Bridget, a high school assistant 
principal, illustrates how one administrator might approach two teachers with different goals and 
how an administrator might take account of a variety of factors when setting TGDC goals. 

Bridget explained that because she was new to the building, she did not know much about 
an English teacher named Becky, who was due to be evaluated. Bridget was very neutral about 
Becky going into the evaluation and began her work with this teacher with an open mind. Becky, 
in turn, was well prepared and at ease with the TGDC online platform. Bridget did not have a 
specific goal and instead followed Becky’s lead, using the TLF to identify areas where Becky 
could enhance her already strong lesson plan. Early on, Bridget realized that Becky had many 
strengths as a teacher but that she was not reaching all of her students. So Bridget’s goals for the 
TGDC cycle became focused on how to nurture English language learners (ELLs) by attending 
to questioning techniques and paying careful attention to student grouping arrangements to 
facilitate a safe, open environment for all students. 

In a physical education (PE) teacher’s case, Bridget had some knowledge of the teacher, 
his teaching, and his needs for improvement. Just as in Becky’s case, Bridget did not have a 
standing relationship with Bruno, the PE teacher. Because Bridget lived in the community in 
which she worked, however, she had heard many negative things about Bruno. Bruno had a 
reputation as being a tough teacher who failed a lot of students. Bridget’s principal asked her to 
conduct the TGDC with Bruno because Bridget did not have a personal history with Bruno and 
because Bridget was the most patient of the assistant principals. Bridget was very committed to 
having a clean slate with Bruno. She was committed to being fair and open-minded with him. 

She learned about his classroom very quickly, however, through her role as parent and student 
problem solver; she received many telephone calls and complaints about Bruno’s teaching. 
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These complaints were particularly urgent and impassioned because PE was a graduation 
requirement. When Bridget began the TGDC activities with Bruno, he did not have a lesson plan 
and did not know how to work with the online TGDC platform. Bridget’s goals for Bruno were 
to “develop a more student-friendly syllabus, class environment, just everything. I wanted for 
him to really plan with the students in mind and the students’ levels and abilities.” 

Bridget, like many other focus administrators, set different goals for different teachers. 

As Table 4 shows, Bridget had substantively different goals for these two teachers despite the 
common underlying goal of improving students’ experiences in the respective classrooms. There 
are some similarities and differences in the factors that appeared to shape Bridget’s goals. For 
both teachers, Bridget noted her background knowledge about the teacher as well as each 
teacher’s preparedness for the TGDC cycle. In light of her knowledge of Bruno, Bridget tried to 
develop a goal that was responsive to his current level of practice. In Becky’s case, lesson plans 
took the place of background information, so Bridget set her goals using the plans Becky 
presented. 

LAUSD emphasized that the TGDC should be responsive to teachers. Goal setting is only 
one aspect of being responsive; however, we saw evidence that administrators were setting goals 
with teachers that were specific to the teachers’ needs. 


Table 4. Goals and Factors in Goal Formation for Two of Bridget’s Teachers Evaluated in 
2013-2014 


Goal/factor 


Goals 


Factors 

Knowledge of teacher 


Specifics the teacher 
brought to the 
interactions with the 
administrator 

Administrator’s prior 
experiences with the 
teacher 


Teacher 


Becky 

Nurture ELLs 
Improve questioning 
Use groups well 

Almost no knowledge 

Strong initial plans and need to help 
more students achieve in the teacher’s 
classroom; teacher at ease with 
TGDC technology; teacher’s turn to 
be evaluated 

No experiences 


Bruno 

Develop student-friendly syllabus 
Improve classroom environment 
Plan with students in mind 

Knowledge of teacher by reputation 
and knowledge passed from principal 
and teacher’s file 

No initial plan; teacher uncomfortable 
with TGDC technology; teacher 
evaluated owing to a history of 
complaints and previous poor 
performance 

Experiences included Bridget having 
to talk with Bruno, parents, and 
students about Bruno’s policies and 
students’ performance in his course 
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Goal/factor 


Teacher 


Becky 

Bruno 

School’s goals 

None mentioned 

PE is a required course that many 
students fail, and parents and students 
complain a lot 

History of the 
administrator in the 
school 

Member of the surrounding 
community 

Member of the surrounding 
community 

Administrator’s 
professional values 

ELLs often not well supported 

ELLs often not well supported 


Strategies Administrators Used to Improve the Processes 

As articulated across this report, administrators found the time demands of TGDC 
burdensome and challenging to manage on a day-to-day, week-to-week basis. They also found 
that certain problems came up repeatedly and began to desire strategies to address these 
problems consistently. For example, a number of administrators noted that low trust or anxiety 
was a barrier to having productive interactions around the TLF. To deal with this, administrators 
developed strategies across the cycle to decrease anxiety and increase trust. Other administrators 
noted that they repeatedly failed to meet teachers right after a lesson, decreasing the impact of 
their feedback. In response, they developed several strategies to ensure timely meetings with 
teachers. The strategies that administrators reported were idiosyncratic and varied, but they 
seemed to address five foundational challenges endemic to supporting adult learners in 
improving their professional practice: (a) efficiency, (b) scheduling, (c) transparency, (d) trust, 
and (e) improvement. 

As mentioned previously, the issue of efficiency came up regularly; administrators had 
too much work to do and too little time to do it. Increasing efficiency was related to a second 
issue: scheduling. Even if someone can do something quickly, urgent issues, such as a problem 
with a parent or raccoons in the garbage, can always come up, making it challenging to keep 
TGDC appointments. There were also the two related issues of trust and transparency. Trust was 
critical for administrators and teachers to work together. Teachers needed to be able to be 
learners and feel that they were being treated fairly and with care. To the extent that teachers had 
a relationship with their administrators characterized by honesty, respect, and kindness, trust was 
likely shared. Trust is also related to transparency. When teachers understood the process and 
what was expected of them, this supported transparency in the evaluation process. Finally, focus 
administrators noticed ways that the TGDC process could be improved. Most improvement 
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strategies were intended to help the teacher or administrator explore the teaching more, 
differently, or in more powerful ways. 

Following is a list of strategies that administrators used to complete TGDC activities by 
addressing these five foundational issues: 

• Strategies across all components of the cycle 

• Do walk-throughs for 2-10 minutes as a regular part of your day (trust, 
transparency). 

• Have teachers share their experiences in the TGDC cycle with each other (trust, 
transparency). 

• Reassign a teacher to a different administrator if that will help the teacher engage 
the process better (trust). 

• Use only the language of the rubric (transparency, improvement). 

• Train teachers on the rubric and have them co-observe (improvement, 
transparency). 

• Set up rough standards across teachers about how long to spend on pre- and 
postconferences and stick to them (efficiency, transparency, scheduling). 

• Work with administrators who observe beginning teachers for other state and 
local programs so that lessons observed under the TGDC count for other required 
beginning teacher programs (efficiency). 

• Encourage everyone in the building to get trained on the TLF to develop a shared 
sense of quality teaching (transparency, improvement). 

• Preobservation conference strategies 

• Hold preobservation conferences with two teachers at a time who are working on 
similar issues (efficiency, transparency). 

• Move to the other side of the principal’s desk to be side by side with the teacher 
(trust). 

• Go to the teacher’s classroom for the conference (trust). 
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• Solicit teachers’ input about which elements to focus on (trust, improvement). 

• Plan lesson together (trust, improvement). 

• Provide templates for essential features of successful lessons in advance (trust, 
improvement, efficiency). 

Lesson strategies 

• Video record lessons on teachers’ iPads so that teachers can privately screen their 
own teaching (transparency, improvement). 

• Create video and/or audio recordings of observed lessons (with permission of the 
teacher) so that more accurate scripting can be done, (transparency) 

• Have the teacher identify things on which he or she would like the administrator 
to provide feedback (trust, improvement). 

• Do not reschedule unless there is an emergency (trust, efficiency). 

• For informal observations, see two to three classes a day for 10-15 minutes 
(scheduling). 

• Do a small amount (1 day per week for 1 hour) of co-observing with two 
administrators (transparency, accuracy of ratings). 

• Take a very good script to refer to during the postconference (transparency, trust, 
improvement). 

Postobservation conference strategies 

• Provide the transcript of the lesson to the teacher in advance of the 
postobservation conference and allow the teacher to verify its accuracy (trust, 
transparency, efficiency). 

• Look together at student data to supplement lesson reflection (improvement). 

• Rate the lesson together with the script and rubric while sitting side by side at the 
computer (trust, transparency, efficiency). 
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• Review the evidence on the last screen of the online rating tool together element 
by element to see what evidence supports each (improvement). 

• Ask teacher to reflect on how the conference went (trust, improvement). 

• Have the conference right away or within 2 days of observation (efficiency, 
improvement). 

• Solicit teachers’ rejoinders to the ratings and maintain openness to changing 
ratings based on teachers’ responses (trust). 

Strategies across teachers 

• Develop checklists of critical issues or questions to ask (improvement, 
efficiency). 

• Bring cases to administrative team meetings to discuss rating and management 
issues (improvement, accuracy). 

• Set weekly goals for the administrative team regarding the number and types of 
observations, adjusting schedules as needed to make attaining these goals possible 
(efficiency, improvement). 

• Use an aligned tool to observe the teachers who are not on cycle so they receive 
feedback on the same types of things on which they will be evaluated in the future 
(improvement, efficiency, trust). 

• Use a second observer if the teacher is feeling unfairly targeted (transparency, 
trust). 

• Select two TGDC elements as a school and focus all your school’s professional 
development on those elements, tying in the TGDC work and associated 
professional development (efficiency, improvement). 

• Provide professional development for the whole faculty that is directly linked to 
focus elements in TLF (improvement, efficiency). 

• Make linkages between the TGDC and other district initiatives explicit 
(improvement). 
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• Schedule a substitute principal on a number of days designated for TGDC work at 
the school (efficiency, scheduling). 

• Enlist others to contribute to TGDC duties (efficiency, scheduling). 

Changes in Administrators’ Use of the Teaching and Learning Framework Over Time 
Summary of Findings 

In this section, we describe the types of changes evident in administrators’ thinking when 
using the TLF over the practice year (2012-2013) and first year of consequential implementation 
(2013-2014) of the TGDC. 

• Some administrators’ use of rubric language during the think-aloud and stimulated 
recall process increased. 

• Some administrators’ scoring approaches changed over time, and references to the 
TLF rubric were more evident in their reasoning processes. 

• Some administrators’ comfort and confidence improved. 

Background 

We conducted a think-aloud activity with focus administrators twice, during the practice 
year and again during the first year of consequential implementation. This activity asked 
administrators to watch a 10-minute clip of a lesson, align evidence from the lesson transcript 
with three TLF elements, and then rate each element. Researchers watched and listened to the 
alignment and rating process while administrators thought out loud. After ratings were 
completed, we asked follow-up questions (stimulated recall) to better understand why the 
administrator gave the ratings he or she did. 

In this section, we analyze the think-aloud data of focus administrators across both years 
of the TGDC. We examine changes in their use of the TLF. The findings in this section come 
from a limited set of data; additional cleaning, coding, and analyses are ongoing. 

Changes in the Use of Teaching and Learning Framework Language 

We found some changes in the language used by focus administrators during think-alouds 
and rating rationales for videotaped lesson excerpts. Maria, for example, used more rubric 
language in 2013-2014 when asked to clarify and justify her ratings. In response to the question 
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“How did you decide what kinds of questions were being asked?” during the practice year, she 
said, “The questions were mostly what, how many — they are not open-ended questions. The 
teacher said ‘You are close’ but didn’t expand.” Maria talked about the evidence she observed 
but didn’t connect the evidence to the rubric or use the rubric language to explain her reasoning. 

During the first year of consequential implementation, Maria said that she wanted “to see 
if the questions are open-ended, if the questions invite further deep knowledge and thinking, if 
there’s following up. The student said 4, tell me why 4, why not 5. Give them that pause.” 
Maria’s reasoning used more TLF language, and she also added her interpretation of the TLF and 
examples of how she understood the rubric. 

When being asked to justify her ratings during the practice year, Bianca, an elementary 
school principal, said, 

The reason I focused on questioning is because I see a lot of whether or not students 
understand is based on questions. If you look at Standard 2, the students are clear about 
the objectives; they’re able to state it in their conversations. 

Bianca referred to the rubric and talked about learning expectations being clear to 
students but did not use the rubric language. In 2013-2014, however, she used specific words 
from the rubric to explain her reasoning: 

I didn’t see what standards she was addressing. Overall learning expectations were 
unclear to students. I know the standards but the kids may not know that clearly. Overall 
learning expectations were not clear. Was it that the students were going to identify odd 
versus even? What about ELL? [Classroom interactions] lack appropriate challenge and 
support, like at the end, lining up with their little cards, did the kids understand the 
concept? Was that sufficient? [Reading the rubric aloud about support], I am wondering 
whether they understood the 100s chart. 

Learning expectations, appropriate challenge and support, and standards are all terms from the 
TLF rubric. As compared with 2012-2013, Bianca integrated the rubric language into her 
reasoning in 2013-2014. 
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Changes in Rating Approach 

Some administrators’ approaches to aligning evidence and rating changed between the 
practice year and the first year of consequential implementation. During alignment in 2012- 
2013, Ron looked at the elements he needed to rate and then went through the transcripts to 
select and align evidence. In 2013-2014, he started by reading the descriptions of the specific 
elements. Another change in his rating approach was articulating his understanding of the rubric 
before going through the transcript: 

When I hear the word norm, I am thinking about behavior, verbally, proximity, and 
physical behaviors. And I am also thinking about learning expectations, could be 
protocols in terms of the instructions. Heading, dates, things related to instructions, not 
just one person speaks at a time, say please and thank you. Cognitively busy places — I am 
thinking about Bloom’s taxonomy, how rigorous or challenging the work is. I look at 
protocols or procedures the teacher has in place and the types of questions or activities 
the teacher prepared for the students. That’s what I get from cognitively busy places, 
which is close to being passive, being observers. 

During the practice year, Ron tried to match the evidence with the four rating categories 
holistically, rather than following the alignment process. Instead, Ron read the descriptions of the 
rating categories one by one to decide whether they matched: 

I read the rubric, the element and look at the criteria; I start with the highest one: [reading 
the rubric] OK he’s not Highly Effective. OK I don’t think it’s clear. I think it’s going to 
be Developing, but I will look at Ineffective. The expectations were read to the students 
but were not explained. 

During the first year of consequential implementation, Ron followed a longer and more 
careful process. He went through the rubric sentence by sentence and reviewed the evidence 
from the video against the descriptions of the rating categories to make his decision: 

Overall learning expectations are clear to all students and consistently related to 
standards. Again, standards is the last word, it should be included in the first sentence the 
teacher utters at the beginning of the lesson. Whatever they are doing, whatever the 
outcome is should be connected to the standards. Usually it’s at the beginning, but for 
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this lesson, I see it to be very weak, too general as an objective, so I will not count that as 
evidence for Culture of Learning. 

During the practice year, when Ron was asked to clarify his reasoning to the researchers, 
he analyzed the rubric descriptions and justified his ratings with rubric language, similar to the 
first-year think-aloud excerpted earlier. This type of reasoning, however, did not appear in his 
practice-year think-aloud, suggesting that as Ron continued to gain experience conducting 
observations, he needed less prompting to engage in deeper reasoning surrounding the content of 
the rubric. 

Focus administrators also referenced specific evidence from the lesson more often during 
their first-year think-alouds as compared with their practice-year think-alouds. Some of them 
relied more heavily on the TLF when explaining their ratings. For example, in 2012-2013, Roy, 
an elementary school principal, explained why he gave a particular rating: 

I just gave it Effective. In my mind the questions I remember hearing, if something rings 
true and effective immediately, I go back to it, but it just doesn’t ring true to me. . . . The 
clutter of the classroom, the culture of expectations, just wasn’t there. 

His response sounded like he was relying on some internal or personal criteria to arrive at 
a rating. It was not clear how (or even whether) the criterion of ringing true corresponded to the 
rubric. Roy did not reference specific evidence from the video or rubric language. 

In 2013-2014, Roy read the descriptions of the Effective and Developing categories and 

said, 

I see [the video] as more toward the Effective; I’m in between Developing and Effective, 
but reading through the rubric, I think it’s closer to Effective. I am looking at the 
interactions: there are a lot between teacher and students, but not among students. There 
are not a lot of opportunities for students to show what they know besides raising their 
hands and telling the teacher. So that kind of has me between the two [ratings]. But 
technically I think it’s Effective. 

This response showed an increased reliance on evidence from the video, along with a 
heightened attention to the language of the rubric. Roy used the rubric when trying to distinguish 
between 2 score points; the word technically indicated that he was referring strictly to the rubric 
to make his final decision. 
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Two other examples illustrate how administrators’ use of the rubric changed since the 
practice year of TGDC. When asked to explain “how did you decide what level of expectations 
the teacher had?” Mary provided specific evidence observed in the video to justify her rating, 
without referring to the rubric or explaining how the evidence supported her decision: “He made 
sure that even the kids who didn’t have a book were able to share versus saying, ‘Why don’t you 
have your book?”’ 

In the first year of consequential implementation, Mary approached rating differently, 
focusing on the differences between the rubric definitions of the Developing and Effective 
categories: 

Ultimately, she wanted all children to be able to stand on either side of the room with a 
card that had either an odd or even number on it; I don’t think those activities support that 
goal. [Reading the descriptions of Developing and Effective:] Overall learning 
expectations are clear to all students and consistently related to standards. If the standard 
is [that] students will be able to understand odd or even numbers, I wasn’t clear if she’s 
relating back to that standard. 

Jason showed a similar change. When asked how he decided whether the activities were 
rigorous and appropriate for the students in 2012-2013, Jason’s responses focused on prior 
knowledge and his own understanding of rigor: “I looked at seventh [grade?] standards in my 
head, my prior knowledge. Rigorous because [this is] not an easy assignment. Geometric shapes, 
which I know is challenging for students.” By the first year of consequential implementation, 
Jason’s process had evolved to carefully read the rubric at various levels (i.e., Developing, 
Effective, and Highly Effective) when trying to decide between two rating categories: 

[Reading the descriptions of Developing and Effective:] I did think they were there. 
[Reading the description of Highly Effective:] The all there is a gray area for me. I didn’t 
have a chance to probe all student minds. Had she had other techniques that allow 
mandatory response, like cue cards where every kid showed an answer at the same time, I 
would lean toward Highly Effective. 

Changes in Administrators’ Comfort Level 

We found that, when using the rubric to rate instruction, some administrators’ level of 
comfort and confidence improved, helping them to better use the rubric in the way it was 


RM-15-12 


32 



C. Bell et al. 


Understanding Consequential Assessment Systems of Teaching 


intended. In 2012-2013, some administrators filled in a lack of evidence for certain elements of 
the TLF with assumptions. In 2013-2014, administrators were more comfortable acknowledging 
that the evidence was not present. For example, when trying to decide if there was instructional 
differentiation during the practice year, Lana made assumptions about the teacher’s behavior 
without offering supporting evidence: “I can only assume there was differentiation when he was 
picking his groups — 1, 2, 3; you guys, 1, 2, 3.” In the first year of consequential implementation, 
Lana admitted that she couldn’t determine the level of differentiation given the evidence 
available: 

I’m not sure she did any real differentiation for the subgroups. You couldn’t tell who 
were [in] the subgroups because questions were one to one to one. I’m a little foggy on 
that. Usually, when you’re in a classroom, you can tell who the subgroup is: ELL 
students, kids needing extra help, students with behavioral problems; some sort of 
something, you can see. Because questions were closed and only allowed one-word 
answers, you didn’t see the differentiation by subgroup. 

Though we can only speculate, it is possible that having practiced and conducted more 
observations, some administrators became more comfortable with the constraints of observing a 
single lesson. They understood that they were supposed to record the evidence and rate the 
lessons based only on the evidence collected. This might indicate that administrators improved 
their abilities to use the rubric objectively and rate instruction based solely on what they 
observed. 


When Did the Process Work Well? When Did It Go Poorly? 

Summary of Findings 

• Teacher self-reflection was frequently cited as a determining factor in the success of 
the TGDC process. 

• Administrators suggested that teachers who were open to feedback were more likely 
to engage with the TGDC process. 

• Quality of relationships between administrators and teachers was cited as a reason 
that observations went well and as a reason that they went poorly. 
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• The TGDC cycle was a source of stress for some teachers, and this stress shaped the 
TGDC cycle. 

• Administrators felt that beginning and veteran teachers were equally likely to have 
positive experiences with TGDC; almost no beginning teachers were thought to have 
negative experiences. 

• Few administrators nominated a teacher’s specific instructional role (e.g., general 
education, special education) as a reason that the process did not go well, although 
two suggested that it was difficult to evaluate special education teachers using the 
TLF. 

• A handful of administrators nominated teachers’ difficulty navigating the My 
Professional Growth System (MyPGS) as the primary reason the TGDC process was 
time consuming. 

Background 

Whereas the previous section focused on how administrators’ use of the TLF changed 
over time, this section examines how the observation cycle unfolded with individual teachers. 
This investigation gives us insight into how administrators thought about the factors that 
promoted success and failure for teachers completing TGDC in 2013-2014. This section is based 
on analyses of data collected during the observation cycle interviews with focus administrators. 
Focus administrators walked us through two cases — one in which they thought the teacher would 
describe the TGDC process as a positive experience and one in which the teacher would describe 
it as more negative. 

Asking administrators about positive and negative cases helped us answer the question, 
What facilitates (or inhibits) the TGDC process? In analyzing focus administrators’ rationales, 
we looked at three broad categories of attributes: (a) attributes of the teacher; (b) attributes of the 
administrator, including the relationship they established with the teacher; and (c) attributes of 
the TGDC process (e.g., the shift toward using objective evidence). These categories were not 
necessarily mutually exclusive; a teacher’s willingness to reflect on his or her practice may have 
been a result of the positive climate established by the administrator, or it may have been a result 
of the teacher’s belief that the TGDC was a worthwhile endeavor. Thus, where relevant, we have 
drawn connections across these categories. It is also important to note that the findings presented 
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are administrators’ perceptions of teachers’ reactions; reasons for the success or failure of the 
process might differ if teachers themselves were interviewed. 

When Did the Process Go Well? 

Focus administrators provided many rationales for teachers’ positive experiences with the 
TGDC. Overall, administrators were far more likely to ascribe positive experiences to teacher 
attributes than to attributes of administrators or the TGDC process. That said, across our sample 
of focus administrators, all three categories of attributes were seen as influencing experiences 
with TGDC. Some of the primary themes of focus administrators’ responses are summarized. 

For the sake of comparison, the same themes are presented first in cases of teachers who had 
positive experiences and then in cases of teachers who had negative experiences. 

Teacher openness to change. The teacher characteristic most often nominated as 
positively influencing TGDC experiences was openness to changing their practice. 
Administrators commonly described the TGDC as being predicated on teachers’ willingness to 
use feedback to improve their practice. This foundation facilitated productive preobservation and 
postobservation conferences. It allowed administrators to suggest areas in which teachers could 
improve without teachers feeling attacked. In contrast, teachers’ unwillingness to change their 
practice (or their belief that they are already Highly Effective) undermined conversations about 
improvement. 

When asked why teachers reported positive experiences with the TGDC process, 
openness to change came up repeatedly in our interviews. One principal, Vicki, noted, 

The process is new for everybody. I don’t think any teacher I evaluated this year had 
gone through the process before. I think the teachers were hesitant, nervous, had a 
difficult time receiving feedback, they take everything I said to them, the feedback I gave 
was negative, rather than looking at it as a process. This particular teacher, who also 
started off that way, when I had to go in and make suggestions that I thought would 
benefit her, she didn’t see it that way but willing to try. I waited and came back, there’s a 
complete change. She also noticed it and said so glad you gave me this strategy. She saw 
a difference that made, making her lesson planning easier, the implementation easier. 
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Vicki’s comments echoed sentiments expressed in several of our interviews. For many, 
TGDC represented a real change in the dynamic between administrators and teachers. 
Historically, teachers and administrators did not always have relationships that focused on 
instructional improvement, and receiving feedback made some teachers feel uncomfortable. In 
the case of Vicki’s teacher, although she was initially uncomfortable, the teacher eventually 
opened up to feedback, changed her practice, and, in Vicki’s opinion, improved as a teacher. 

Glen noted a similar improvement when one of his teachers opened up to feedback: 

We created very specific actionable strategies and he followed them. From observation to 
observation, I could see him becoming more comfortable with those strategies. He also 
started sharing some of the things he did in his classroom. When we reflected at the end 
of the year, it was the purposeful questioning where he was Highly Effective, and he 
never thought he would reach that. 

A related teacher attribute that many focus administrators nominated as a source of 
positive or negative experiences was teacher self-reflection, which we conceptualized as distinct 
from willingness to change. In the preceding quotations, the teachers were willing to change their 
practice, but suggestions for improvement came from the administrator. In a smaller number of 
cases, administrators suggested that the teachers who saw this process as positive reflect on their 
practice and take it upon themselves to change. One focus administrator, Maria, described a self- 
reflective teacher: 

She’s very welcoming [toward] any tool that will help her grow as an educator. She’s 
very receptive and reflective. So in our conversations, as we debrief about our 
observations, she was always very positive. . . . When she would score herself — we 
always score her lessons together, she’s very honest about it — yes, I understand this part, 

I need to grow in this area, or this is an area of my strength. That really helped her, as a 
matter of fact, she already set goals for the coming year. 

Another administrator, Donna, also commented on the reflectiveness of one of 
her teachers: 

He’s veteran teacher, if put in a 1-10 scale, probably at a 4. He really struggled with the 
TGDC process initially, but his reflection piece was the most powerful and impressive at 
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the end. He wrote six pages, really analyzed himself, how he taught, the student work, 
all of that, he really took it to heart to make sure he made progress in his teaching. 

In each case, the teachers took steps to improve their instruction above and beyond the 
formal requirements of the TGDC cycle. In the former, the teacher laid out TGDC-based goals 
for the upcoming year. In the latter, the teacher wrote a long, thorough reflection. 

Teacher experience and grade/subject. The teachers who had positive TGDC 
experiences varied in years of teaching experience. Administrators were just as likely to 
nominate beginning teachers as they were to nominate mid-career or veteran teachers. Some of 
the teachers were new to their schools but not new to the profession. As mentioned earlier, 
administrators often selected teachers who were new to the profession, new to the school, or had 
not been evaluated in several years to engage with the TGDC process. 

Some focus administrators nominated teaching experience as a reason the process went 
well, although having several years of experience and having minimal experience were both used 
as rationales. Some focus administrators commented that more experienced teachers were more 
likely to be comfortable with their instruction and thus were more successful in engaging with 
the TGDC process. In contrast. Heather suggested that beginning teachers were more likely to 
have positive experiences with the TGDC: 

I think it was a lot easier in the sense that these are brand-new teachers. I only went 
through the TGDC with my new teachers. I had six teachers that were all first-year 
probationary teachers. Their affective filters are all really high. The teachers get really 
anxious but the anxiety level is very high. The conversations, everything being laid out, 
that helps them. 

Administrators did not view teachers’ grade levels, subjects, or instructional roles as 
influencing positive or negative experiences with the TGDC. 

Teacher instructional ability. Teacher instructional ability came up in a handful of 
interviews as a reason why some teachers were more successful than others. Bridget, for 
example, said, “The reason I would say this is a positive experience is because she is already not 
developing. She is already proficient. She gets up on herself to leam the process. Knowing that 
makes it easier for her and me.” Also, several teachers were nominated because, though they 
initially were weak in one or more areas, they grew through the process. It appeared that a 
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teacher’s initial skill levels mattered less to administrators than whether the teacher 
acknowledged that there was room for improvement. 

Teacher-administrator relationship prior to the teacher growth and development 
cycle. As discussed previously, focus administrators developed many strategies to improve the 
efficiency, scheduling, trust, transparency, and improvement orientation of the TGDC process. 
Here we focus on the quality of teacher-administrator relationships prior to TGDC. 
Administrators were asked to reflect on their relationships with teachers at the beginning of the 
school year. 

It might be expected that initially close relationships between teachers and their 
administrators led to more positive experiences with the TGDC process; we certainly saw 
instances of this in our data. For example, Anthony described a teacher with whom he was close 
as “always coming to me for advice, consulting for advice. Incredibly dedicated, chose me and 
trusted me in my ability to guide her.” The logic is that, if the teacher trusts the administrator, he 
or she will be open to the administrator’s feedback. Administrators were just as likely to cite 
teachers with whom they had solely professional relationships, however. Sometimes this was 
because the teacher was new to the school so the administrator and teacher had not yet built a 
relationship. Overall, our results suggest that mutual respect is a far more common marker of a 
positive TGDC experience than a close relationship. 

Attributes of the teacher growth and development cycle. Administrators in some 
instances nominated the TGDC process itself as a reason certain observation cycles went well. 
Administrators felt the specificity of the rubrics allowed teachers to focus on aspects of their 
instruction they wanted to improve: 

She learned to be more reflective and learned to use the teaching and learning rubrics to 
align herself to where she wanted to be. She wanted to improve her practice and be 
effective, and the specificity of the rubrics made it easy for her to see what she needed to 
do to put herself in that spot. ( Charles ) 

Portia described similar value (for the teacher) in having a clear set of instructional rubrics: 

She would say that setting the expectation, the Teaching and Learning Framework, and 
knowing that there were certain strategies that we needed to see or that she needed to do 
to score Highly Effective or Effective helped her focus and/or improve her practice. 
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Administrators also felt that the TGDC process provided time and space for 
teacher reflection: 

She [the teacher] would probably say that she likes the structure even though she didn’t 
fully understand the purpose of everything. It allowed her some time to do some 
reflection. I think she would probably also say it’s a better way to provide feedback to 
teachers than whatever we’ve used in the past. (Bridget) 

These quotations reinforced sentiments on the strengths of the TGDC. However, they 
also suggested that the TGDC is particularly well suited to teachers who want to improve their 
practice. The TGDC provided space for reflecting on instruction and a clear road map to the 
aspects of their instruction that could be improved. 

When Did the Process Go Poorly? 

Administrators also nominated reasons why the TGDC process went poorly with their 
teachers. Similarly, many of the rationales focused on qualities of the teacher themselves. It was 
common that the administrator’s prior relationship with the teacher impacted the TGDC process 
negatively. A small number of administrators also described specific issues that their teachers 
had with the TGDC process. 

Teacher openness to change. While openness to change was listed as the primary reason 
the TGDC process went well, it was also the primary reason the process went poorly. The 
majority of focus administrators commented that teachers’ unwillingness to change their practice 
undermined the TGDC process. Our focus administrators described how teachers who had 
negative experiences refused to consider changing their practice based on TGDC feedback. 

Maria described one of her teachers: 

She’s not open-minded. How can I put it, what she’s been doing for the task for the past 
whatever number of years has been working. If you were to ask her about TGDC, she 
would say the district is squashing us. She doesn’t like change; change is not comfortable 
for her, very much so, it’s very difficult for her. She would say everything in the rubric, 
she’s doing it. Even though you show her the evidence, she will find a way of saying that 
she’s doing everything that’s on the Highly Effective side. 
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Portia described another teacher who simply shut down in the face of recommendations 
to change: 

She was very uncooperative; she had to be given several memos just to meet with us. The 
teaching and learning director finally gave up on her; she requested that I assist her more 
with this one or that one at times when the teacher refused to do her lesson and we had to 
reschedule, and I had to do that one on my own because the teacher was unwilling to 
cooperate and do the observations when the teaching and learning observers were here. 

While a limited number of teachers were unwilling to engage in the TGDC process, 
others participated but struggled to reflect on their instruction and, as a result, developed 
negative impressions of themselves as teachers. Darlene described one such teacher: 

She’s well into her career; she has been given that information or at least that’s what she 
said. She was shocked and surprised that she was not. . . . She was given Developing and 
Ineffective. Her self-image as a teacher — she was aghast. She had no idea. 

The logic of the TGDC process is that, in scenarios such as the preceding, teachers 
respond to these shocks by taking steps to improve their instruction. Our data included examples 
of some teachers who did this and others who used the negative feedback as further justification 
for disengaging from the TGDC process (i.e., completing the required steps but just going 
through the motions). 

Teacher experience and grade/subject. With only one exception, administrators did not 
nominate beginning teachers as negative cases. Instead, administrators nominated relatively 
similar numbers of mid-career and veteran teachers. No discernible patterns were related to 
teacher grade, subject, or instructional roles in negative cases, with the exception of two 
administrators who said that their special education teachers had the most negative experiences 
with the TGDC. 

Teacher-administrator relationship prior to the teacher growth and development 
cycle. In many cases, when focus administrators were asked to describe the background of the 
teachers who had negative experiences, they described previous incidents that had weakened 
their relationships with the teacher. The TGDC process did not damage the relationships but 
represented an opportunity for previous conflicts to reemerge. For example, Charles said, 
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He was a must place about 10 years ago. I’m just finishing my fifth year; I was advised 
when I took over this school by the outgoing principal. They placed him at our school 
thinking that a new environment might help; he was pretty cautious the first couple years, 
but soon old habits set in and there were quite a few problems. 

Elizabeth described a similar situation: “[The teacher] was put in my school because she 
has had difficulties in all of our other schools. She comes after school; she tore up my 
recommendations and threw it in the trash can.” Finally, Vincent told us, 

She has a long history of interpersonal problems with other administrators. I was given a 
heads-up about her ahead of time. It became very apparent early on. It wasn’t just 
instructional issues, but it was with parents and families and with colleagues. There was 
conflict with students about grades. There was a lot of conflict and that trickled down to 
instruction. When you look on a teacher in totality, it’s hard to focus on instruction when 
so much else is going on. And we had to keep removing students from her class because 
the conflict would flare up. The whole class would be upset. 

Perhaps most troubling about this case was that the teacher’s previous history with 
administrators manifested in her classroom climate and instruction, which had a direct impact on 
students. While these kinds of stories certainly did not appear in all of our interviews, or even in 
the majority of them, they occurred commonly enough to warrant attention. In these cases, 
preexisting administrator-teacher problems may have had an adverse effect on the TGDC 
observation process. 

Attributes of the Teacher Growth and Development Cycle. Finally, administrators 
described a number of different aspects of the TGDC process that contributed to negative 
experiences for their teachers. Heather described the TGDC as having introduced anxiety that 
negatively affected her relationship with her teacher: 

I think that the formal process of going through the observation process was intimidating 
to her to begin with. She didn’t seem comfortable yet with the process. She felt like there 
were parts that were missing — was she thorough enough in her responses? Is this OK, 
Heather? She had logistical problems with loading things and would call me frantically. 
She had a lot of apprehension with the pieces. It was constantly overwhelming to her. 
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Bridget noted that the technology component of the TGDC increased the time demand 
and stress associated with the process for one of her teachers: 

The hours, the hours, it was 1-2 hours every week. I had to go and enter things myself for 
him in the platform. His technology skills are so below basic. That already makes the 
process frustrating. 

Ella also commented on the time demands of the process and the stress that they caused 
for her teacher: 

Every time I asked her how she’s doing, she complains about how long it takes, she stays 
up late to type it up. If I wanted to visit her, she would say, “I need to go do an 
observation.” She’s very stressed out about the TGDC process work and the time it took. 

Ella went on to discuss how the time demands and stress appeared to affect her 
relationship with this teacher over time: 

[The relationship] was better before. I don’t know if it’s because I focused on her as an 
instructional leader, and I expect to see changes on instruction in the classroom. Our 
conversations last year were focused on resolving the conflicts with parents. She wants to 
do her own thing. When I try to get her to be on board with what should be happening she 
has excuses for everything. Felt like the relationship was better last year with conflicts 
with parents and teachers, and this year when I focused on instruction it didn’t go well. 

The teachers’ struggles with the TGDC, including time demands, stress, and problems 
with technology, mirrored some of the challenges that administrators themselves faced with 
TGDC implementation. 

Lessons Learned: A Case Study of Two Principals 
Summary of Findings 

In this section, we investigate the experience of two focus administrators. We did not 
select these principals to be representative of all the observers in the study; rather, they were 
selected because their experiences highlight some compelling issues that arise in the context of 
developing and deploying observational assessment systems of teaching. Following is a 
summary of the findings related to the two principals in this case study. We emphasize that these 
issues are not necessarily present for all or even most observers: 
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• The TGDC process was affected by seemingly unrelated issues at school sites. 

• Administrators found the TLF to be well aligned with their instructional aims. 

• Teachers who perceived themselves to be Highly Effective were surprisingly difficult 
to evaluate. 

• Administrators processed conflict differently. 

• Issues of race and class were present in different ways relative to the TGDC. 

• Administrators desired more and different support from superiors. 

• Administrators used the TGDC to accomplish multiple goals at their schools. 

• The TGDC complicated administrators’ work but also helped them manage aspects of it. 

• Administrators created a number of ways to manage the workload of the TGDC, but 
time remained a critical issue. 

• Administrators frequently invoked their sense of mission, especially in trying 
situations. 

Background 

In this section, we offer a multiple-case case study to illustrate some of the issues we saw 
across administrators implementing the TGDC in the district. In other sections of this report, we 
relay information issue by issue; this section of the report provides a different angle by 
conveying the TGDC experienced by individuals to illustrate how TGDC developed over time 
for two administrators and their schools. We selected two principals to study in depth, Anthony 
and Danya, because they represent highly contrasting cases that also have much in common. 

They are, on one hand, extreme cases: Anthony is an enthusiastic user of the TGDC and has 
created strategies and forms to support the teacher evaluation work in ways that give TGDC its 
best possible implementation; Danya felt defeated by the TGDC and was challenged to 
implement TGDC in ways that were meaningfully focused on instruction. At the same time, both 
administrators had their share of recalcitrant teachers; were fairly new administrators; and felt 
they had little support from the district, their faculties, or their colleagues. 

The multiple-case case study revealed six dimensions that were salient to administrators’ 
implementation of TGDC. Each of these dimensions is described in the following case studies 
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and illustrated with quotations from the interview and think-aloud exercises of the two 
administrators. The six dimensions are (a) understanding of the TLF; (b) supports and challenges 
in implementing the TGDC; (c) attending to conflict; (d) perception of teachers, teaching, and 
improving instruction; (e) management of the TGDC; and (f) principal identity. To capture the 
range of administrators’ engagement with the TGDC across the district, we provide portraits of 
two administrators in LAUSD who share much but who also diverge greatly in their views and 
practices. We begin with Anthony, a principal who is a wildly enthusiastic proponent of TGDC 
and an innovator in the way it is used. We then meet Danya, a principal who shares many of the 
same school and community challenges that Anthony faces but whose experience with TGDC 
stands in stark contrast to Anthony’s. 

Anthony: The Teacher Growth and Development Cycle Enthusiast 

As mentioned previously, Anthony views the TGDC with great exuberance. The 
following quotation typifies Anthony’s approach to TGDC: “If this disappears, I will continue to 
do it. To me it’s an incredible data collection system and a great framework for designing lessons 
and improving instruction.” This quotation is just one of many that Anthony offered in a similar 
vein across the 2 years during which he participated in the study. 

Some of Anthony’s enthusiasm for the TGDC may be a function of his identification with 
the instrument chosen for observational assessments. The evaluation of teachers in LAUSD is 
shaped by the TLF to a large degree. The extent to which administrators’ work on teacher 
evaluation is affected by the chosen instrument cannot be overstated; nearly all evaluation 
activities are framed by this instrument. Thus principals’ interpretation and use of the TLF are 
crucial in our understanding of how the TGDC proceeds in LAUSD. Anthony makes frequent 
reference to the TLF even when not prompted to do so in interviews; his conversation is 
frequently peppered with language from the TLF. 

Anthony identifies with the portrait of instruction painted by the TLF — to say that he 
embraces the TLF would be an understatement. Although this conforms to our findings across 
administrators in LAUSD (see, e.g., the section What Went Well), Anthony is perhaps an outlier 
on this issue. He is thoroughly committed to the substance and process of TGDC and says so 
frequently. 

In the following quotation, one senses Anthony’s continual focus on the TGDC as a 
vehicle for improving instruction. His comments return repeatedly to issues in Standards 2 and 3. 
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Where other administrators may focus on the teacher’s demeanor or the climate of respect in the 
classroom, Anthony’s focus is on subject matter instruction. This is demonstrated in his use of 
instructional terms taken directly from the TLF. For example, in his think-aloud responses 
watching a video excerpt of a lesson during Year 2, he said, 

This is where I was documenting that teacher was using Level 2 questions, and I don’t 
know even the rigor of these questions when you’re asking something about. . . . What’s 
the purpose of a question like that? How does that push the issue of even and odd? 
Yesterday I tried to look at Common Core to see what they have on even and odd. When 
you look at the mathematical practices, when you are looking at TGDC it must have 
some [inaudible]. I’m not sure when, this lesson was completely unnecessary. I believe 
the kids knew what even and odd numbers were. But I don’t see, another thing that I find 
that was truly annoying was going through a series of questions asking one even number 
at a time. Is she stretching the time? When you have the digits 0 through 9 and you have 
identified the numbers, she went through one at time. 

This quotation highlights Anthony’s focus on the mathematical content of the lesson he 
rated (odd and even numbers) and the pedagogical decisions the teacher made in conducting the 
lesson. 

Anthony uses the language of the rubric and its ideas. Unlike other administrators we 
have interviewed, Anthony repeatedly returns to the notion that the TLF and the broader TGDC 
effort are fully aligned with other district initiatives: 

All the Standards for Mathematical Practice have been in the California Math 
Framework, 21st Century Skills, have been on the books for a decade. Every time you 
call [TGDC] new, you are strengthening the argument of the contrarians. Something 
called TGDC has teeth in it; it’s new accountability. 

I am selling the district initiatives and pinpointing what is new that is not totally new, it’s 
a different way of doing the same thing. Either the lesson plan or the Common Core or 
the TGDC is something that I talk about — when I have an opportunity to talk at meetings 
or with a teacher. Now that more teachers can do it, it will be more of a two-way 
conversation. Until recently I’ve been doing most of the talking. My practice of taking 
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notes and giving them feedback — I’m taking them through the TGDC cycle and they 
don’t even know it. 

. . .You’re not telling teachers that everything they’ve done in the past is wrong — what 
you’re telling teachers is that all that we have been doing we’ve done already except for 
this little piece that we’re adding on. . . . The other elements were in the old system, 
they’re just now improved. The tech piece is not a problem for them. We need to explain 
how the system has changed, help teachers understand that it’s not really brand new. 

What is different is that you have established what you will be going in to see ahead of 
time. 

From the first year of the TGDC, Anthony frames the evaluation process in terms of 
teacher learning — and compares this to student learning and ways to document and encourage 
learning: 

What I think evaluation is, and what I think growth and development is: When you’re 
evaluating, it’s like summative. The way the system, you are not — although it gives the 
impression that it is formative, it is more summative than anything else. Yes, the process 
is totally different and you negotiate and go back and forth, but in the end for the teacher 
in real life there is very little room to put that growth and development in when you have 
two shots at that thing. Maybe because I didn’t do the second cycle I didn’t get to see 
that. My thing is that if the focus is on helping teachers grow and develop, there should 
be more instances than those two formal cycles. It’s like expecting children to get better 
if we only give them the two end-of-semester assessments. But the fact that I didn’t 
participate in the second cycle — maybe that’s where all of that happens. 

The best example with progress monitoring for our students, I just completed my first 
year here. The district wants to have performance dialogues. But you can dialogue all you 
want, but the better dialogues take place when we collect data all along the way. 

Echoing something we have heard from other administrators, Anthony especially likes 
the reliance on evidence for evaluating teachers: 

When I know what I’m supposed to be doing I sleep well. The TGDC has given me help 
in that. If you think I have made a mistake in that, then let’s look at the evidence. How 
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did I fix that from Cycle 1 to Cycle 2? I had them score from the evidence — you would 
see that teacher sweating. It’s not so hard to rate a teacher with the evidence. Here’s 
where the lesson design and going back and forth, we have a preconceived agreement of 
what we are going to see, bang bang bang. Can you show me the evidence? You would 
see them turning colors. 

In fact, Anthony has developed a sort of protocol for the postobservation conference and 
rating based on what he describes. He transcribes the full lesson and provides the script to the 
teacher for his or her approval before their postobservation conference. Then, during the 
conference, he determines the ratings along with the teacher, adhering strictly to the 
transcription. At each element, they look for evidence in the transcript to support a rating. This 
protocol has a number of virtues in Anthony’s mind: It depersonalizes the rating system, it makes 
the process real and credible to teachers, and it gives Anthony confidence that he is not giving 
capricious ratings. It makes the process feel more factual and less subjective: 

I am ending a year battling things that I could have done better if I had had the correct 
support. I am doing this the way it’s supposed to be done. I have heard that some 
principals in order to avoid problems with their staff assign Effective and Highly 
Effective ratings. These teachers that I assigned Developing, they hate me now, they hate 
me with all their hearts. I have the data to show them. I invited them to look at the rubric, 
and that is an incredibly humbling experience, even if they have the transcripts. It’s not 
the same when you ask the person how would you rate yourself, especially when I’m 
finding the evidence. I’m also open to using taping. I ask them for permission, they say 
yes, and I’m taping everything now. 

Similarly, both to manage the demands of the TGDC and to keep meaningful 
instructional improvement the focus, Anthony has devised another protocol — one that guides 
administrators and small groups of teachers to design lessons together: 

I have tried to find ways to make it streamlined, like in TGDC, I do things in small 
groups instead of one on one and talking one on one. Remember I asked this question in 
the first session? I thought I needed to really interact with them in this lesson design 
process, that’s what I did, in most cases one on one, in some two on one, for K and Grade 
2. We looked at expectations and the elements of the lessons. In that conversation I had 
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an amazing discovery: when you want all the elements present in one lesson, you end up 
with a lesson that is not a lesson, but you end up with something crazy. Show me an 
entire unit and where these lessons are present in an entire unit. If I don’t see all six in a 
single lesson, show me where I’ll see them in the unit. With these two third-grade 
teachers, I wanted some kind of rubric, which evolves from the criteria chart, and you 
need to have students be part of the development process. Even though you know all the 
elements that will be in it, you need to have the students participate in it. For the lesson 
they were going to show me they didn’t have a rubric but they had a criteria chart. We 
decided that’s OK, don’t force the rubric when it doesn’t belong, I can come back to see 
the lesson later when the rubric is in play. This is not very distant from where the 
Common Core wants us to go — no longer looking at individual lessons, but looking at 
entire units of study. 

As part of this protocol, Anthony nominates six focal elements that must be present in a 
unit for teachers to receive high ratings. He has learned that it’s helpful to work together with 
teachers in developing these lessons and making sure that the requisite elements are included. 
Although this is not required in LAUSD, Anthony has made the requirement of six focal 
elements part of the process in his school. He has developed forms to support these planning 
sessions, sequestered time for teachers to work together, and secured funding for teachers to get 
professional development in support of what he considers the essential elements of the 
instructional unit: 

Let’s say that part of all these initiatives requires an exorbitant amount of work, planning. 
I cannot just tell teachers, “you have to plan.” I have injected this time at $70/hour, 
substitute teachers — this is how I can contribute. The other part comes from them. When 
you know that part of the work is to plan and I am requiring now evidence of planning, 
now I have a revolution against me; they complain right and left; do I just not require 
evidence of planning? 

For some of the teachers at Anthony’s school, the TGDC has been a positive experience. 
But he has had serious conflicts at his school — some related to the TGDC and some not — that 
compose the backdrop of the work on instruction. For example, the one teacher who volunteered 
to participate in the TGDC during the practice year (2012-2013) stopped speaking to Anthony as 
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a direct result of their work in the observation cycles. This teacher considered herself to be an 
excellent teacher and was stunned to learn that Anthony rated her teaching as Developing in 
many places. She ended the process — refusing to complete any more of the forms and 
activities — and effectively froze him out. 

Yet Anthony is unmoved by this experience when it comes to the value of the TGDC. He 
is disappointed in the teacher’s reaction but undaunted: 

We had amazing growth, out of 13 teachers. I had to drop one, special education teacher; 
I allowed her to do it in language arts, but it didn’t work. Everyone made these amazing 
[gains] , finally realized that this was real, and they were in shock that they were 
Developing — that was like offensive. Maybe one teacher was found Developing, out of 
the 13 teachers. I know one for sure, but not because of a bad experience, she’s a new 
teacher that is Developing. If I can say that one of my teachers, it’s not because of the 
experience; it has to do with attitude. It was the same teacher who dropped out of the 
[practice] year. Once I gave her the ratings and gave her Developing in some areas, she 
just shut down. She was probably overall Effective, but the one that has the most issues 
with everything else. 

This year, Anthony continues to have challenges from United Teachers Los Angeles 
(UTLA) and from teachers to whom he refers as contrarians: 

I think everyone benefited from it. I have a very strong group of contrarians who were 
against it, but they knew they had no choice. What differentiates the positive from the 
negative is the attitude. Categorizing all these things as new is a mistake. When they 
found themselves developing, they complained about my professional development. She 
began to question me when I thought it was unnecessary, that’s Teacher B. 

Anthony faced serious challenges on multiple fronts at his school last year, some of 
which were exacerbated by the introduction of the TGDC, yet his enthusiasm for the TGDC has 
been unflagging. The union at his school was rumored to have told teachers not to upload their 
lesson plans, and the teachers with whom he was working productively and intensively on 
aspects of instruction specified in the TLF were intimidated by some fellow teachers. The 
teachers were later unwilling to present their instructional improvements at a faculty meeting for 
fear of making the others look bad. Additionally, parents picketed Anthony’s school demanding 
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action against a teacher — action that had in fact already been taken but that Anthony could not 
disclose. Although this was distressing to Anthony and reflected badly on him and his school, he 
felt the parents were right to demand high standards for their children. 

Anthony does not feel these conflicts have been mitigated by district support — in fact, 
Anthony has made multiple comments that district personnel above him have made his work 
more difficult: 

There is a huge dysfunction, every time that I go to my principals’ organization, EPO. Dr. 
Deasy [the district superintendent] is the keynote, I go out of the meetings pumped up: 
let’s rock the system, make the mistakes you have to make — man, he’s talking to me. I go 
to the local district with my boss and she’s openly against central. I don’t have her as my 
boss. We’re going to make mistakes along the way. Disruptive innovation is exactly what 
Dr. Deasy asks us to do. I’ve done it here, with this infrastructure of change, with all the 
memos I thought it wouldn’t be a bumpy road; man, was I mistaken. 

Anthony finds these setbacks distractions from his work. We will see in the next case that 
Danya takes similar setbacks in a very different way: Where she feels defeated by them, Anthony 
feels a renewed sense of purpose. Both Anthony and Danya express a strong sense of mission 
about their work, and they both reference this frequently following the recounting of a severe 
challenge in their work. Anthony describes, “I’m a man on a mission. I wouldn’t work in a rich 
area; I always want to work in poor neighborhoods.” It is this sense of mission that Anthony 
invokes when he expresses frustration with teachers who are reluctant to change or district 
personnel who do not support his work. 

Danya: The Rule Follower 

Danya provides an interesting contrast to Anthony: They share a great deal but in other 
ways are polar opposites. Like Anthony, Danya talks frequently about a keen sense of mission to 
serve families in poor neighborhoods, and she returns to this theme when talking about 
distressing situations in her school: “I know how it feels to be a LAUSD student. I grew up in 
this neighborhood.” 

Danya shares this comment while discussing a conflict she is having with a teacher. This 
conflict comes to define Danya’s experience with the TGDC, and she ultimately loses her 
position at the school after a rocky year with her faculty, marked by skirmishes about teacher 
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performance in a number of different areas. One teacher has volunteered to be evaluated this 
year, and ironically, it is this teacher, Angel, that Danya believes brought on her downfall. 

Danya had a very different experience with TGDC during the practice year. She was in 
her 10th year as an assistant principal at a different high school, where she rated one volunteer 
teacher — an experience that felt uneventful and productive: 

I had a very nice volunteer teacher who piloted it with me and she was open to helping 
me navigate the platform because she was more tech-savvy and I helped her think about 
ways to engage students in more feedback, deliver more purposeful questions. 

From the outset, Danya has had serious concerns about the amount of time that the 
TGDC takes, especially in combination with her operational duties: 

If that’s all you had time for, it can be fulfilling, for both teachers and administrators a 
light bulb will go off in their heads about their practice. Good learning process for the 
individual; in the perfect world, like writing a diary for yourself, you’d become a better 
person for it. You’d only be doing that for it to be working, not pep rallies, not alcohol 
being smuggled into the girls’ bathrooms, not pedophiles in the building, not a bloody 
nose. 

Like other administrators, Danya notes that the TGDC is valuable for certain aspects of 
teachers’ work but not as relevant for others: 

The teacher I worked with was tardy often, had a million excuses. I’ve been an 
administrator for 10 years and tardies take up a lot of time. That improved because I had 
to handle her administratively in addition to EGDC. EGDC doesn’t address that. TLF 
focuses on everything but housekeeping issues. Housekeeping is like turning in syllabus, 
grade book, attendance records. EGDC focuses on everything but housekeeping stuff, but 
a lot of times that’s what we are busy with. EGDC is awesome if you have all the time in 
the world to focus on the lessons. We’ll get a call from a parent to check to make sure the 
student is in class and if the teacher hasn’t submitted attendance records we can’t tell the 
parent — we have to send the police out. 

Things change dramatically for Danya when she moves to a new high school as principal. 
In this new school, the TGDC becomes associated with other points of conflict she experienced 
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in establishing herself as the school leader. Danya has tried to impose some order around issues 
that, to her, are central to healthy school functioning; to her faculty, however, these issues seem 
arbitrary and unrelated to student learning. For example, Danya has insisted that teachers sign in 
and out of the school every day and that their pay be docked if they are tardy. These new 
practices have alienated many veteran teachers on her staff, creating a contentious atmosphere. 

Unlike Anthony, whose focus is consistently on rigorous content learning for students, 
Danya sees teacher evaluation as being as much about following rules as it is about learning: 

DANYA: To intentionally bend the rules because someone’s going to break the rules — I 
can’t do that. A group of teachers were hell-bent on getting rid of me, and they 
succeeded. “This new principal is so strict; she’s docking our pay if we’re tardy; she’s 
making us sell healthy snacks at Halloween.” 

interviewer: Would you ease up on any of this next time? 

DANYA: No, I can’t pick and choose. We check the dress code, common stuff. That just 
didn’t work out for me. I’d go back to my normal leadership style. I’d change if my 
principal recommended it. I came into this because I want to help students. Why am I 
making teachers spoiled brats? A teacher’s workday is eight to three; that is so light. 

They wanted to sneak off campus before three. 

Danya focuses on elements of the TLF that are less concerned with instructional content. 
When asked what elements she finds most critical for effective teaching, Danya said, 

Standard 4, Component 4c: ethical conduct and compliance with district and state 
regulations. Not being sarcastic with students so that you don’t shut down their affective 
filters. Getting to your work station on time, yes. Respectful treatment of students, yes. 
Not handing in grades or attendance on time, not really. 

Danya’s comment about having grown up in the school neighborhood and knowing “how 
it feels to be a LAUSD student” contrasts with her perception of Angel’s privilege and disrespect 
for poor students of color. Danya gives low ratings to Angel for respect and climate in addition 
to “writing her up” — reporting her to the central office — for the sarcastic way that Angel speaks 
to students. In Angel’s mind, this casual banter with students is building rapport; to Danya, this 
conduct is unprofessional and disrespectful: 
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Angel is the one who volunteered and said she comes from the corporate world: I got 
evaluated all the time, come any time. So when I came to visit her classroom formally 
and informally, I was surprised that some of the things she was saying were 
inappropriate. I went to the district to run it by them, and they said she should be written 
up. When I gave her that feedback she got really defensive, and it’s been really tense 
since then. She said some really inappropriate things that were not culturally 
responsive — things that are not OK. 

One of the questions she posed, “Would you like to read this article? No? You want to 
stay ignorant?” For African American kids, that is a very big insult. You didn’t give wait 
time. 

There were three other [incidents]. You can let it go one or two times; once I saw a 
pattern, I had to intervene. She said she used the word intentionally, that ignorance is a 
choice, she saw no possibility that it was inappropriate. The district’s mission is to 
believe in the dignity and worth of all students, and this is definitely working against that. 

The differences between Danya and Angel were never overcome. In fact, Danya reported 
that she has manipulated TGDC ratings as a way to make peace with teachers with whom she has 
been in conflict. Recall that Anthony manages TGDC implementation by devoting resources and 
developing structures for teachers to focus on student learning; Danya, in contrast, uses the 
TGDC as a way to repair relationships with her teachers: 

I would try to go by the book if all is well and good, but in trying to earn the trust of new 
staff I’d go extra easy on them. But that’d be the right thing to do if I didn’t want strife. 
Then it’s not based on evidence but on the situation. I let a lot of things go at the end. 

Where Anthony finds that relying on evidence improved the teacher evaluation process, 
Danya anticipates ways in which teachers will manipulate or ignore evidence: 

The whole thing about an evidence-based evaluation system goes out the window when 
you are dealing with personalities and you’re trying to save your job. It doesn’t work 
anyway. The purpose of evidence-based evaluation goes out the window. 

Danya has turned to the district for a number of supports: At the behest of Angel, she 
invited a TLC and a central office staff member with expertise in TGDC to take part in 
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observations and ratings. In Danya’s eyes, these district personnel are not helpful and even 
undermine her work: 

The district backed the teachers up, from my director to my local district superintendent. 
It’s more hurt to my ego; I worked so hard to get here. A bunch of teachers want to place 
blame on the principal. It’s happening everywhere. They’re moving principals all over. 
I’m very disillusioned with the way this was done. I’m hearing through the grapevine that 
Dr. Deasy has zero tolerance; he thinks that principals are part of the problem. He hosted 
new principal support meetings and I always felt rejuvenated; he always said to deal with 
administrators who were not on your team. He also wanted teachers to adhere to 
guidelines. . . . 

The TGDC framework is so strict. You have to really work intentionally to make your 
lesson fit everything. To me if you’re going to work really hard to be a darn good teacher, 
no matter where I come from, older, younger, everybody has a basic respect for authority. 
When I realized that I wasn’t getting the backup. If I didn’t break policy, they wouldn’t 
take me out. But the district was not backing me up, my boss kept questioning me. Do I 
really want to go through this again? 

For both Anthony and Danya, it’s the strong teachers who believe themselves to be 
excellent that are the most challenging to work with. For both of these administrators, conflict 
has ensued when the teachers received ratings lower than Highly Effective. The teachers have 
not been open to suggestions for improving their instruction, feeling insulted and defensive. And 
interestingly, both of these principals have been genuinely surprised and dismayed to observe 
aspects of instruction in strong teachers’ classrooms that were not just flawed but deeply 
problematic: 

At the same time [Angel] has a stellar reputation, she picked up a class that no one 
wanted to teach, so has a great can-do demeanor, her strengths are that the kids receive 
her well, they don’t run and tell on her, their culture is to be compliant, or they think it’s 
funny, or they see her as cool maybe. She has a great reputation and she is a great 
teacher. So we’re working on how to balance that. She’s refusing to acknowledge her 
rating because she doesn’t agree with the ratings I issued her, so we’re in the middle of 
battling it out. I don’t know how this one’s going to hold. That whole relationship has 
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switched between her and I. It’s hard when you have to engage in that personality issue 
of it. I look at the rubric and here’s the evidence I found. It has to meet all the evidence of 
Highly Effective. Even on her reflection piece, she doesn’t take ownership of what she 
can do differently in instruction. She’d have students peer edit the papers, but I wanted to 
get her to say that I would put an anchor paper up; I would model think- alouds; I would 
give students auditory input about where this academic concept would come into play. 

She doesn’t talk about how she can change her delivery of instruction. She didn’t get as 
many Effective or Highly Effective [ratings] as she thought [she would]. 

Issues of race and class have hung over Danya’s relationship with Angel and with some 
other teachers at the school. Danya has mentioned on more than one occasion that Angel comes 
from the corporate world, an elite college on the East Coast, money. Angel has been interviewed 
in the newspaper as being a proponent of charter schools, implying to Danya that she has 
reservations about public schools. In a particularly tense postobservation conference, Angel 
mentions that she believes that Danya rated aspects of her Advanced Placement Statistics class 
poorly because Danya did not understand the content. 

Danya perceives that people treat her like hired help, rather than as a person of authority 
and knowledge, because she is Latina: 

I think that’s a huge part of why I failed here. All the former principals here were male 
and White. . . . They saw this younger Mexican American principal and weren’t going to 
take me seriously. A lot of the teachers who complained about me were White. I kept my 
cool and most people would not have. It was pretty blatant with one teacher. Her 
disagreement with me: No, I am not going to do that. No, I don’t think you are right. She 
chose other evaluators to come in. She expected Rachel, the head of TGDC, to evaluate 
her because she didn’t trust me. 

Danya’s interpretation stands in contrast to Anthony’s view. Although Anthony is also 
Latino, he does not feel that issues of race and class are at play in his relationships with his 
teachers — except to the extent that some teachers hold low expectations for students who are 
poor and Hispanic: 

There is an attitude, not only here, when you serve in these depressed communities that 
these children cannot learn.” That’s something that I stated very clearly when I heard 
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them for the first time: They did not belong here and those people who say them should 
not teach here. These comments come from Hispanic teachers about Hispanic students! I 
have a structure that they will use for conferences because the parent conferences were 
otherwise two words: hi, bye. Our parents are submissive, didn’t complain. I have sent it 
in the newsletter, and at the entrance to the school, for questions parents are supposed to 
ask, in our weekly bulletin and in phone messages. 

It’s about the attitude of the teachers toward poor families. Not between myself and 
teachers. But having this attitude — comments like, “There’s nothing I can do for this 
child.” 

In summary, the TGDC is overlain upon numerous issues, ostensibly separate from 
teacher evaluation, with which principals and teachers are grappling. The cases of these two 
administrators illustrate that the TGDC can be mediated — and complicated — by the relational, 
instructional, and institutional work that transpires in schools. We also see how the interplay of 
principals’ dispositions, school-wide challenges, and teachers’ idiosyncrasies impacts TGDC 
implementation. 

Report Summary 

Focus administrators shared many insights during the 2013-2014 school year. In the 
following, we offer a bulleted, high-level summary of our findings. 

Implementation Summary 

• Administrators were responsible for an average caseload of five teachers. 

• Administrators used a variety of strategies to select teachers to engage in the TGDC 
process. In addition to selecting teachers new to the profession, administrators 
nominated teachers who were struggling, who volunteered, and who were new to the 
school but not the profession. 

• On average, observers spent approximately 7 hours per week on all TGDC-related 
activities. 

• An average TGDC cycle took approximately 12 hours of an administrator’s time. 
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Strengths and Weaknesses of the Teacher Growth and Development Cycle 

• Focus administrators expressed a positive view of the overall TGDC process and 
considered it to be a useful system to support educators. 

• The strength of TGDC most frequently discussed was the TLF. 

• Some administrators reported that they noticed improvement in technology use as 
well teachers’ attitudes in 2013-2014 as compared with the practice year. 

• More than half of the focus administrators reported that what didn’t go well was the 
huge time demand; a majority of the administrators identified that as the primary 
weakness of TGDC. 

• Technical difficulties were still one of the most frequently mentioned weaknesses of 
TGDC and reasons for things not going well this year. Administrators suggested 
teacher training and additional technical and human resources for administrators. 

• Administrators wanted opportunities to discuss TGDC with colleagues. 

Administrators’ Use of the Teacher Growth and Development Cycle Process With 
Teachers 

• Focus administrators used information from other interactions with the teacher (e.g., 
walk-throughs) to inform some aspects of the TGDC cycle. 

• Many administrators tailored the logistical specifics of the cycle to the teacher (e.g., 
length of pre- and postobservation conferences, location, and number of cycles). 

• All focus administrators seemed to script lessons. 

• Administrators varied in their willingness to depart from the prescribed TGDC 
process. 

• Focus administrators used the TGDC for both evaluation and improvement purposes 
but overwhelmingly focused on, sought out resources for, and thought about how to 
support instructional improvement. 

• Administrators’ goals for teachers varied across individuals and over time. Goals 
were sensitive to the specific contexts of the people and schools involved in the 
TGDC cycle. 
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• Administrators developed many strategies to improve the implementation of the 
TGDC cycle. The strategies addressed five foundational issues: efficiency, 
scheduling, transparency, trust, and improvement. 

Changes in Administrators’ Use of the Teaching and Learning Framework Over Time 

• Some administrators’ use of rubric language during the think-aloud and stimulated 
recall process increased. 

• Some administrators’ rating approaches changed over time, with references to the 
TLF being more evident in their reasoning processes. 

• Some administrators’ comfort and confidence improved. 

When Did the Process Work Well? When Did It Go Poorly? 

• Teacher self-reflection was frequently cited as a determining factor in the success of 
the TGDC process. 

• Administrators suggested that teachers who were open to change were more likely to 
engage in the TGDC process. 

• The quality of relationships between administrators and their teachers was cited as a 
reason that observations went well and as a reason they went poorly. 

• The TGDC cycle is a source of stress for some teachers, and this stress can shape the 
TGDC cycle. 

• Beginning and veteran teachers were just as likely to be nominated as having positive 
experiences with TGDC; almost no beginning teachers were nominated as having 
negative TGDC experiences. 

• Few administrators nominated a teacher’s instructional role (e.g., general education, 
special education) as a reason that the process did not go well, although two 
suggested that it was difficult to evaluate special education teachers using the TLF. 

• Teachers’ difficulty navigating the MyPGS system was nominated by a handful of 
administrators as the primary reason the TGDC process was time consuming. 
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Appendix A. How Was This Report Prepared? 

This report is based on data collected from a number of sources between July 2013 and 
July 2014. A group of 38 focus administrators were followed for the first consequential year of 
the TGDC. As described in the Overview section of this report, they filled out online 
questionnaires, participated in interviews, described in detail how they were working with 
specific teachers, and completed think-aloud exercises while viewing video-recorded clips of 
classroom practice. 

Here we describe each of these instruments and the methodological approach we used to 
analyze the data. The instruments themselves are included in Appendixes B and C of this report. 

Focus Administrator Surveys 

During the 2013-2014 school year, we administered four different surveys to a total of 38 
focus administrators, including three bimonthly short surveys and a longer survey at the end of 
the school year. The surveys were all administered online through SurveyMonkey. After the 
initial invitation, a reminder was sent to those who did not respond within a week. Focus 
administrators were provided with an opt-out option to block any future survey invitations. 
Survey items were developed based on established measures, drawing on guidance from 
LAUSD’s central office staff. For all of the surveys administered, data were first cleaned, and 
then descriptive statistics were conducted. 

Bimonthly Surveys 

The first bimonthly focus administrator survey was sent out to all focus administrators on 
November 11, 2013, via SurveyMonkey. The survey was developed collectively by the research 
team and included seven short questions. It mainly focused on checking administrators’ progress 
and experiences using the TGDC observation process in the 2013-2014 school year thus far and 
polled administrators for a good time to conduct the first telephone interview. Of the 38 
contacted, 36 focus administrators responded to the survey. This instrument is found in 
Appendix B. 

The second bimonthly focus administrator survey was sent out to focus administrators on 
February 7, 2014, via SurveyMonkey. The survey was developed collectively by the research 
team and included seven short questions. The main focus of this survey was the role of the 
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TGDC process in administrators’ jobs and schools and the support they most needed with it. Of 
the 38 contacted, 34 focus administrators responded to the survey. This instrument is found in 
Appendix B. 

The third bimonthly focus administrator survey was sent out to all focus administrators 
on May 27, 2014, also via SurveyMonkey. The survey was developed collectively by the 
research team and included 1 1 questions. It mainly focused on the time administrators spent on 
TGDC-related tasks and other teacher evaluation activities. Of the 38 contacted, 31 focus 
administrators responded to the survey. This instrument is found in Appendix B. 

Los Angeles Unified School District Observer End-of-Year Survey (2013-2014) 

The end-of-year survey was sent out to all administrators on July 11, 2014, via 
SurveyMonkey. The survey was developed collectively by the research team with input from the 
district and included 12 questions. It mainly focused on administrators’ self-assessment of their 
job efficacy and preparation, the allocation of their time, their strategies for coping with various 
aspects of their jobs, their impressions of the TLF, their opinions about the observation 
component of the TGDC, and their understanding and use of the TGDC process. Of the 38 
contacted, 30 focus administrators responded to the survey (Table Al). This instrument is found 
in Appendix B. 


Table Al, Survey Schedule and Response Rates 


Survey name 

When delivered? 

No. of questions 

Response rate (%) 

Bimonthly Survey 1 

November 11, 2013 

7 

95 

Bimonthly Survey 2 

February 7, 2014 

7 

89 

Bimonthly Survey 3 

May 27, 2014 

11 

82 

End-of-Year Survey 

July 11,2014 

12 

79 


Focus Administrator Interviews 

Four interviews were conducted with focus administrators over the 2013-2014 school 
year: the initial, observation cycle, end-of-year, and think-aloud interviews. 

Initial interview. Of the 38 focus administrators, 28 were interviewed by research team 
members at the end of 2013 and in early 2014. The interviews took approximately 60 minutes, 
and all of them were conducted over the telephone. The interviews were audio recorded with 
interviewees’ consent. The initial interview protocol was developed collectively by the research 
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team and included eight open-ended questions. In this interview, administrators were asked to 
describe the teachers they were evaluating this year: their basic information, strengths and 
weaknesses, and anticipated response to feedback during the TGDC process. The interview also 
included questions about how TGDC fit into administrators’ other work, their thoughts on what 
went well and not so well with TGDC, what support they need to be successful, and their big 
goals for the school year. The interview questions are found in Appendix C. 

Observation cycle interview. To better understand the specifics of how focus 
administrators worked with teachers around the TGDC, we gathered semistructured interview 
data about two teachers for each administrator. In spring 2014, all focus administrators selected 
two teachers — one who likely would say that he or she had a positive experience and one who 
would likely say that he or she had a more negative experience with the TGDC. Interview 
questions elicited the administrator’s and teachers’ experiences with each observation cycle, 
number of formal and informal observations conducted, observational logistics (e.g., duration of 
preobservation conferences, location of conferences), teacher’s background and history at the 
school, and goals for that teacher for the 2014-2015 school year. Thirty-two observation cycle 
interviews were conducted: 20 with principals, 8 with assistant principals, and 4 with 
administrators who held coaching or instructional director positions. 

End-of-year interview. Of the 38 focus administrators, 29 were interviewed by research 
team members and the interviews were audio recorded with interviewees’ consent. The 
interviews took approximately 60 minutes, and all of them were conducted over the telephone. 
The end-of-year interview protocol was developed collectively by the research team and 
included 14 open-ended questions. It mainly focused on administrators’ leadership style and 
approach, their views of good teaching, their experiences evaluating teachers using the TGDC, 
their impressions of the TGDC, and their plans for the TGDC cycle in 2014-2015. The interview 
questions are found in Appendix C. 

Think-aloud interview. Think-aloud interviews were conducted with 26 out of 38 focus 
administrators in summer 2014. They took approximately 1 hour, and most sessions were 
conducted over the telephone with the aid of GoToMeeting, which enabled researchers to share 
video-recorded lessons and observe administrators as they aligned evidence and rated the lesson 
using the TLF. 
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We used the same think-aloud protocol that the research team had developed collectively 
in 2012-2013. Focus administrators were asked to watch a 10-minute video of classroom 
instruction, with transcripts of the video provided. Then they aligned one piece of evidence to 
each of the three focus elements we selected, rating the lesson using the TLF rubric, as they 
normally did during an observation cycle. In this process, they were asked to verbalize their 
thoughts as much as possible. When they completed rating, a stimulated recall portion was 
conducted with specific follow-up questions about how the focus administrator thought about 
specific scales (e.g., why a certain score could not be higher or lower than what the administrator 
assigned; how he or she used and evaluated the evidence collected). At the end of the session, we 
also asked them to provide feedback on the process of rating and to share their experiences using 
the rubric. The think-aloud protocol is found in Appendix C. 

Data Analysis 

Many analyses were conducted during the second year of the study, and additional 
analyses are ongoing. Some description of analytic procedures appears in the body of this report. 
More detailed descriptions follow. 

It is important to note that the researchers adhered to certain conventions while analyzing 
the data and writing this report. Aside from the four questionnaires described earlier, this report 
relies primarily on qualitative data collected from focus administrators. Because administrators 
have very limited time and data collection was burdensome for them, we tried to restrict 
interviews to 1 hour in duration. This meant that some administrators did not get to all of the 
questions during their interview sessions. To deal with this unevenness, we quantify findings 
with words such as most, many , and a majority to convey aggregate responses from all or almost 
all of the participants. Similarly, we use words such as most and many to convey instances where 
more than two-thirds of the focus observers responded in a particular way. 

Survey analyses. Survey measures were developed based on existing surveys and on 
guidance from educational leadership faculty members and principals currently working in the 
field. For all of the surveys administered, data were first cleaned, and then descriptive statistics 
were calculated. 

Think-aloud analyses. Think-alouds were discussed regularly at weekly project 
meetings prior to formal analyses. Members of the research team took as close to verbatim 
transcripts as they could during the think-aloud interviews. In the Changes in Administrators’ 
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Use of the Teaching and Learning Framework Over Time section of this report, we analyze the 
data from a limited set of think-aloud sessions. We only include administrators who participated 
in the study and completed think-aloud exercises during both years (n = 15). We focus on 
describing the types of changes that emerged from longitudinal data on the same administrator. 
These data were analyzed by reading through the think-aloud notes and creating an initial set of 
codes with which the notes were then coded. There were three initial codes: changes in 
observers’ use of TLF language, changes in observers’ scoring approach and use of evidence, 
and changes in observers’ comfort level in using the TLF to score. Representative examples of 
each code were drawn to illustrate the results in the report. 

Interview analyses. Interviewers took verbatim notes during all interviews; these notes 
were checked against the recorded interviews for accuracy. In the Strengths and Weaknesses of 
the Teacher Growth and Development Cycle section of this report, we discuss the analysis of 
data from Questions 5 and 6 of the initial interview to learn what worked well and not so well 
with the TGDC and what resources observers needed. The data were analyzed by reading 
through all of the interview notes and creating an initial set of codes on three major categories of 
information we wanted to explore. The data were then coded into the themes we describe in the 
report. 

In the same section of this report, we also discuss analysis of data from Question 4 of the 
end-of-year interview: “At this point, what do you think are the strengths and weaknesses of the 
TGDC?” These data were analyzed by reading through all of the interview notes and creating an 
initial set of codes on the strengths and weaknesses of TGDC; the interviews were then coded 
into the themes we describe in the report. 

For the section Administrators’ Use of the Teacher Growth and Development Cycle 
Process With Teachers, observation cycle interviews were analyzed by reading through all of the 
interviews and creating an initial set of codes. Five initial codes were shared in data analysis 
discussions and further refined. These codes captured which teacher the focus administrator was 
describing (teacher); details about how the preobservation conference, lesson observation, and 
postobservation conference went (interactions); details about where conferences happened; how 
long lessons were (logistics); the teacher’s reactions to the TGDC experience (teacher reaction); 
and the goals the focus observer had for the teacher in 2013-2014 (teacher goals). These codes 
were then grouped to produce the themes we describe in the report. 
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Case study. A multiple-case case study was prepared for inclusion in this report (Yin, 
2009). We selected two principals for whom we had rich data, including the instruments 
described earlier and in-person observations conducted during the 2013-2014 year as part of an 
extension of our original study design. The research team considered a number of administrators 
for this multiple-case case study and selected two based on convenience and their interesting and 
contrasting experiences with the TGDC. 

Qualitative data analyses were conducted for this multiple-case case study using all 
interview and think-aloud data collected throughout 2013-2014. We used Dedoose software for 
open coding and then sorted and grouped codes in a grounded theory fashion (Saldana, 2013; 
Strauss & Corbin, 1998). Themes were noted and developed further in analytic memos; 
overarching categories of administrator reasoning were illustrated with representative quotations 
from each of the two individuals featured. 

Writing 

This report was collaboratively written by the UCAST research team. Team members 
worked to triangulate data from different sources, where appropriate. Quantitative and qualitative 
analyses were coordinated on weekly conference calls and through the writing process. 
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Appendix B. Focus Administrator Surveys 
First Bimonthly Survey 

Thank you for participating in the UCAST study and agreeing to complete this brief 
questionnaire about your experience using the TGDC observation process in the 2013-2014 
school year. The purpose of this questionnaire is to see where you are in the TGDC process and 
to inform us about scheduling our first interview with you. 

The questionnaire should take about 5 minutes to complete. As with all the data we are 
collecting from you in this project, your responses will be confidential and your identity will not 
be revealed to anyone. Your candid responses will help us better understand and improve the 
training and ongoing supports available to observers. Thank you for your participation! 

1. Altogether, how many teachers are going through the TGDC cycle in your school this year, 
including those working with an assistant principal, instructional specialist, etc.? 

Number of 
teachers: 

2. How many teachers are you working with on the TGDC cycle this year? 

Number of 
teachers: 

3. Up to this point, how many, if any, formal observations have you conducted during the school 
year 2013-2014? 

Formal 

observation(s): 

4. Up to this point, how many, if any, informal observations have you conducted during the 
school year 2013-2014? 

Informal 

observation(s): 
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5. Are you working in a new role this school year? If yes, please explain: 

• Yes 

• No 

My new role this year is: 


6. Are you working in a new school/building this school year? If yes, please explain: 

• Yes 

• No 

My new school/building this year is: 


7. We would like to schedule our first phone interview with you for some time before your 
winter break. We expect that it will take no more than 30-60 minutes. 

Please check all weeks that might work for you. 

• November 18-22 

• November 25-27 

• December 2-6 

• December 9-13 

• December 16-20 

Thank you! 


Second Bimonthly Survey 

Thank you for participating in the UCAST study and agreeing to complete this brief 
questionnaire about your experience using the TGDC observation process in the 2013-2014 
school year. The purpose of this questionnaire is to learn more about what role the TGDC 
process plays in your school. 
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The questionnaire should take about 10 minutes to complete. As with all the data we are 
collecting from you in this project, your responses will be confidential and your identity will not 
be revealed to anyone. Your candid responses will help us better understand and improve the 
training and ongoing supports available to observers. Thank you for your participation! 


1. Please rate, on a scale from (faster) to (slower), the following statements: 



Faster 

Somewhat 

faster 

About the 

same 

Somewhat 

slower 

Slower 

Compared to last year, scripting of 
the lessons is: 






Compared to last year, the process 
of aligning evidence to elements is: 







2. Please rate, on a scale from (harder) to (easier), the following statements: 



Harder 

Somewhat 

harder 

About the 

same 

Somewhat 

easier 

Easier 

Compared to last year, using the 
Truenorthlogic platform is: 






Compared to last year, conducting 
a single observation cycle is: 







3. What percentage of your weekly time (on the job), on average, is spent on all evaluations (e.g. 

10 %, 20 %)? 


4. Prior to the TGDC, what percentage of your weekly time (on the job), on average, was spent 
conducting any type of evaluation (e.g., 10%, 20%)? 


5. In terms of support, what are your needs at this point with respect to the TGDC? Select all that 
apply. 

• Training on the TLF framework 

• Having coaching conversations with teachers 

• Time management strategies 

• Technical support using the TNL platform 

• Having conversations with/working with other principals on the TGDC implementation 
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Other (please specify) 


6. To what degree does TGDC support or hinder your efforts in the following aspects of your 
job? 



Hinders 

Has no 
impact 

Somewhat 

supports 

Strongly 

supports 

Setting your school-wide goals for the year 





Providing instructional guidance and/or support to 

ALL teaching staff 





Promoting best practices among ALL teaching staff 





Reviewing lesson plans and providing meaningful 
feedback to ALL teaching staff 





Providing meaningful feedback to ALL staff based on 
observed teaching 





Developing and implementing strategic and school 
improvement plans 





Promoting a positive school climate 





Promoting student learning 





Counseling poorly performing teachers out of the 
profession 





Working to implement the Common Core State 
Standards (CCSS) in your school 





Implementing the ELL initiatives in your school 





Managing relationships with UTLA 






7. In addition, we are interested in gathering information about how the TGDC process fits in 
with everything that goes on in a school. As such, we hope to have a small number of schools 
serve as focus schools this spring. Focus schools will allow us to see the TGDC 
implementation process in person. We will be in touch at a later date if we’re able to work 
with you as a focus school. 


Would you be interested in learning more about being a focus school this spring? 

• Yes 

• No 

We very much appreciate your feedback. Thank you very much for your time and for 
allowing us to learn from you. 
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Third Bimonthly Survey 

Thank you for participating in the UCAST study and agreeing to complete this brief 
questionnaire about your experience using the TGDC observation process in the 2013-2014 
school year. The purpose of this questionnaire is to learn more about how you spend time on 
TGDC and teacher evaluation. 

The questionnaire should take about 5-10 minutes to complete. As with all the data we 
are collecting from you in this project, your responses will be confidential and your identity will 
not be revealed to anyone. Your candid responses will help us better understand and improve the 
training and ongoing supports available to observers. Thank you for your participation! 

1. How many teachers did you evaluate this year using the TGDC? 

Number of 
teachers: 

2. How many teachers are in your school altogether? 

Number of 
teachers: 

3. How many hours did it take you to complete one full formal observation cycle for a single 
teacher this year, including all aspects of the cycle (pre- and postobservation meetings, 
scheduling meetings, entering data into the MyPGS platform, tagging, and submitting final 
scores)? 

Hours: 

4. Did the amount of time you spent completing one formal observation cycle vary from teacher 
to teacher? 

• Yes 

• No 

5. If you answered “yes” to Question 4, what’s the shortest amount of time you spent on one 
formal observation cycle for a teacher? 

I Hours: j I 
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6. If you answered “yes” to Question 4, what’s the longest time you spent on one cycle for a 
teacher? 

Hours: 

7. On average, how many hours per week did you spend on all tasks related to formal and 
informal TGDC observations this school year? 

Average 

hours: 

8. On average, how many hours per week did you spend on all tasks related to teacher evaluation 
(e.g., formal and informal observations, network meetings about teacher evaluation, additional 
observations, staff professional development on teacher evaluation)? 

Average 

hours: 

9. To help us better interpret your responses to Questions 7 and 8, on average, about how many 
hours a week do you devote to all tasks related to your job? 

Average 

hours: 

10. Here are some areas where I got better in using the TGDC: 


11. Here are some areas where I still struggle with TGDC: 


Thank you! 
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Los Angeles Unified School District End-of-Year Survey (2013-2014) 

Thank you for your participation in the second and final year of the UCAST study. In this 
survey, we ask you to reflect on your experience with the TGDC observation process this school 
year. Your reflections will help us understand what needs to be changed or supported as well as 
what is working. Your individual responses to these items will be confidential and your identity 
will not be revealed to anyone. The survey should take no more than 20-30 minutes. Thank you 
for your participation. 


Ql. How effective do you feel at each of the following aspects of your work as an administrator? 



Not 

effective 

Somewhat 

effective 

Mostly 

effective 

Very 

effective 

Supervising staff and students 





Managing student discipline 





Managing budgets and resources 





Conducting classroom observations and walk-throughs 
(including TGDC) 





Providing instructional feedback to support teacher 
improvement 





Planning and facilitating professional development for 
teachers 





Communicating with parents 





Meeting with district personnel 





Working with local community members or 
organizations 





Completing paperwork related to compliance issues 






Qla. If there are elements of your work that are not accounted for in this list, please indicate 
them below: 
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Q2. How well trained have you been on each of the following aspects of your work as an 
administrator? 



Not well 
trained 

Somewhat 
well trained 

Mostly well 
trained 

Very well 
trained 

Supervising staff and students 





Managing student discipline 





Managing budgets and resources 





Conducting classroom observations and walk- 
throughs (including TGDC) 





Providing instructional feedback to support 
teacher improvement 





Planning and facilitating professional 
development for teachers 





Communicating with parents 





Meeting with district personnel 





Working with local community members or 
organizations 





Completing paperwork related to compliance 
issues 






Q2a. If there are elements of your training that are not accounted for in this list, please indicate 
them below: 


Q3. On average, approximately how many hours per week do you spend on each of the activities 
below? 


Note: Please round up to the nearest hour and avoid the use of decimals or minutes (e.g., 4, 5, 6). 
If you do not spend time on a given activity, please write “0.” 
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Supervising staff and students 


Managing student discipline 


Managing budgets and resources 


Conducting classroom observations and walk-throughs (including TGDC) 


Providing instructional feedback to support teacher improvement 


Planning and facilitating professional development for teachers 


Communicating with parents 


Meeting with district personnel 


Working with local community members or organizations 


Completing paperwork related to compliance issues 


Other 



Q3a. If you entered a number other than “0” for the “other” category above, please specify: 


Q3b. The total number of hours you entered above for Q3 is: 


Q4. On average, how many total hours do you usually work in a week? 


Q5. With this year of TGDC implementation completed, to what degree do you agree with the 
following statements? 



Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

1 feel confident in my ability to use MyPGS, the 
online observation tool. 





1 feel confident in being able to collect objective 
evidence of instruction. 





1 feel confident in my ability to align evidence. 





I understand the standards, components, and 
elements in the LAUSD Teaching and Learning 
rubrics. 





I understand the differences between the levels 
of performance (highly effective, effective, 
developing, and ineffective) in the Teaching and 
Learning rubrics. 





1 feel confident conducting preobservation 
meetings with teachers. 





1 feel confident conducting postobservation 
meetings with teachers. 
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LAUSD’s training prepared me adequately for 
my observation work this year. 





1 feel confident in my ability to help all teachers 
improve their teaching. 






Q6. What is your impression of the Teaching and Learning Framework? 



Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

The Teaching and Learning Framework covers 
important domains of teaching. 





The view of instruction underlying the Teaching 
and Learning Framework is similar to my own 
view of instruction. 





Observing using the Teaching and Learning 
Framework has helped me think more critically 
about teaching. 





The Teaching and Learning Framework is a 
valuable tool for talking to my teachers about 
their instruction. 





The Teaching and Learning Framework will be 
more likely to improve the quality of teaching in 
my school than the STULL process did. 





The Teaching and Learning Framework meets 
the teaching and learning needs specific to my 
school. 





I am comfortable with this observational 
instrument. 






Q7. To what degree do you think the observation component of TGDC is useful for the 
following purposes? 



Not at all 
useful 

A little 
useful 

Mostly 

useful 

Very useful 

Identifying or rewarding strong teachers 





Teacher improvement/development 





School improvement 





Teacher dismissal 





Increasing the amount of time I spend on 
teaching and learning 





Creating a common vision of excellent teaching 





District improvement 





Preparing teachers for working with the 

Common Core State Standards (CCSS) 
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Q8. In your current position, to what extent do you agree with each of the following statements? 



Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

I can handle the time demands of the job. 





I can handle the paperwork required of the job. 





I can maintain control of my own daily 
schedule. 





I can motivate my teachers. 





I can generate enthusiasm for a shared vision of 
the school. 





I can manage change in my school. 





I can handle the discipline of students in my 
school effectively. 






Q9. In your experience, to what degree are the following groups of teachers more challenging 
than most teachers to observe and evaluate? 



Not more 
challenging 

A little more 
challenging 

Somewhat 

more 

challenging 

A lot more 
challenging 

Very experienced teachers. 





Teachers older than the principal. 





Teachers with whom a principal has had a 
conflict. 





The strongest teachers in the school. 





Teachers who are not personally liked. 





Teachers outside a principal’s subject matter 
expertise. 





Special education teachers. 





Teachers working at grade levels less familiar 
to the principal. 





Teachers who have a different style than the 
observing principal. 





Teachers who find change challenging. 





Teachers who are reflective. 





Teachers who are satisfied with the quality of 
their current teaching practice. 
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Q10. Reflecting back on the 2013-2014 school year, to what extent do you agree with each of 
the following statements? 



Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

I was able to implement the TGDC with all 
teachers who were supposed to be evaluated. 





I was able to handle the daily demands of the 
job. 





I was able to focus on the quality of instruction 
and learning in each classroom. 





I was able to carry out the operational policies 
and procedures that are necessary to manage my 
school. 






Q1 1. To what extent do you agree with the following statements? 



Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

I am satisfied with my work. 





I am happy with my current school assignment. 





I am glad to be working in my district. 





I think about working outside of public 
education. 





If I could do it all over again. I’d choose the 
same profession. 





I’d consider moving to a different position in 
my district. 





I’d like to stay in education but move to a 
different school or school district. 





I am able to accomplish the things I hope to at 
work. 





I feel I have control over what I do at work. 





I am satisfied with my life. 






Q12. For the 2013-2014 school year, what percentage of full-time K-12 classroom teachers you 
worked with would you put in the following categories, based on your overall opinion of their 
teaching performance? (Please note: Percentages should equal to 100. If no teachers fit the 
category, please write “0.”) 


Highly effective teachers 


Effective teachers 


Developing teachers 


Ineffective teachers 



Thank you for your participation! 
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Appendix C. Focus Administrator Interviews 
Initial Interview 

UCAST: Initial Interview for Focus Administrator 2013-2014 

Good [morning/aftemoon], my name is ., and I am a member of the 

UCAST research team. Thank you for taking time out of your busy schedule to participate in this 
study. As you may remember, the goal of this research is to understand how observers learn to 
use observation protocols in teacher evaluation systems. 

Today, we want to learn from you about your experiences with the TGDC cycle this year. 
The interview should last no more than 60 minutes. If you need to take a break at any point, just 
let me know. 

Before we begin, I want to remind you that your identity will be kept completely 
confidential and the results from this work will not be used in any way to evaluate you or anyone 
participating in the research. If you agree, I will audio record today’s interview just so that I can 
refer back to it when completing my notes. We will keep the audio recordings until the data 
cleaning and analysis are complete. The recordings then will be destroyed. If this is all OK with 
you, I will go ahead and turn on the recording in just a moment. 

Housekeeping Questions 

• Do you have any questions before we get started? 

• Is it all right with you if I start recording now? 

• [if “yes,” start the recorder ] 

Interview Questions 

1. We are doing some checking of the data we have on file, so could you remind me 
how long you have been an administrator? (assistant principal? principal?) 

a. Have you been in LAUSD all of that time? 

b. Have you always been at the XXX (elementary, middle, high school, etc.) level? 

2. Are you at the same school as you were last year? 

a. If yes, this year, how have teachers been selected for evaluation in the TGDC? 
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b. If no, what do you know about how the teachers to be evaluated through the 
TGDC cycle were selected this year? 

3. Can you go through and list of all the people that you are working with this year and 
briefly tell me about them? You can refer to them in whatever way is comfortable — 
first and last name, last name, initials. Everything you tell me is confidential, and I 
will turn the names into pseudonyms when I am typing up the interview. We are 
interested in basic information: their grade or subject level, their strengths and 
weaknesses, and also something about how you expect them to respond to feedback 
you give them as a part of the TGDC process. [Once they finish the description of a 
person, ask...] 

a. Follow-up: That was a lot of helpful detail on each person. I’m wondering if we 
could go back and have you tell me a little bit about each one of them as a 
learner? 

b. Follow-up: How do they learn best? What motivates them? What makes learning 
hard for them? 

4. When you think about all the teachers you personally need to observe this year, how 
are you thinking about spreading your time and effort across them? 

If they spend time and effort disproportionately among teachers, ask them what 
influences their decisions to do that. 

a. Follow-up: How does doing TGDC this year fit into other parts of your work? 

b. Possible follow-up: What strategies are you using to help fit the TGDC into your 
other responsibilities? 

c. [For those who participated in last year’s pilots ] Follow-up: How does this year’s 
experience with the TGDC compare to last year’s experience with it? 

5. I would love to hear your thoughts about what has gone well and not so well with the 
TGDC this year. 

a. [Follow up if they don ’t explain ] What makes you feel that way? 

b. [For those who were just trained] Tell me about how you feel the training has 
prepared you for doing the TGDC cycle. 
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6. Based on your experiences thus far, what resources or supports do you think would be 
helpful for doing the TGDC cycles this year? 

a. Follow-up: What resources or help are available to you, and in what ways have 
you used them? 

7. Changing topics to a bigger-picture perspective on your work, can you tell me about 
your big goals for this school year? 

a. Follow-up: Why are those so important now for you? 

b. How does the TGDC process fit or not fit into those overall goals? 

c. How are people at your school responding to the TGDC? 

8. Can you tell me about some of the other big things happening in the district and how 
they impact the goals you have for this school year? (If people have to drop a 
question, this is the one to drop.) 

Sign-Off 

That was my last question. Thank you for participating in our study. Do you have any 
questions for me? 

[Answer their questions, and/or if you cannot, make a note to forward the question to the 
appropriate person ] 

[Include an update about what the next steps of the project will be] 

Thanks again for participating. Your feedback is immensely helpful, and we greatly 
appreciate it. 

Completion Work 

1. Associate the categories below to the description of each teacher the administrator 
nominated. If more than one applies, write both. [YOU SHOULD NOT SAY these aloud 
TO THE PARTICIPANT, AND THIS SHOULD BE DONE BASED ON EVERYTHING YOU 
LEARNED ABOUT THE TEACHER FROM THE ADMINISTRATOR] 

This teacher is [a] . . . 

a. Self-starter, constant learner, needs very few suggestions for improvement 

b. Learner, responds positively to suggestions for improvement 

c. Able to learn, does not take suggestions for improvement easily 
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d. Does not take suggestions for improvement at all, should not be teaching 

e. Does not meet any of these descriptions 

2. Upload the sound file to Sharepoint so we have it. Delete it from your computer. 

3. Convert to pseudonyms. If your person doesn’t have one, make one up and add it to 
the pseudonym list on Sharepoint. For the teachers listed, make up pseudonyms so 
that our write-ups do not have teacher’s real names. Use the same initials as the 
original name, e.g., real teacher Courtney Bell becomes Caroline Burke. 

4. Put your interview notes onto Sharepoint using pseudonyms. 

Observation Cycle Interview 

UCAST Focus Administrator: Year 2 Observation Cycle Interview 

Thank you for agreeing to help us understand how administrators learn to use observation 
protocols in teacher evaluation systems. Today, we are going to try to understand how you have 
used TGDC with your teachers this year. The whole session should last no more than 60 minutes. 
If you need to take a break at any point, just let me know. 

In our previous conversation/e-mail exchange, we asked you to select two teachers’ files 
that we will take a close look at. We asked you to select one teacher who you think would report 
that the TGDC process was positive and one teacher who you think would report that the TGDC 
process was negative. Today my questions will be focused on your experiences working with 
these two teachers. We will spend approximately 30 minutes for each teacher. 

Like last time, you should use whatever language you want to refer to the teacher (e.g., 
initials, first names only). Regardless of how you refer to the teachers, we will keep everything 
you say confidential, and the results from this work will not be used in any way to evaluate you 
or anyone participating in the research. If you agree, I will audio record today’s interview just so 
that I can refer back to it when completing my notes. We will keep the audio recordings until the 
data cleaning and analysis are complete. The recordings then will be destroyed. If this is all OK 
with you, I will go ahead and turn on the recording in just a moment. 
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Teacher 1 

1. Let’s start with the first teacher you selected — the person whom you thought would 
report a positive experience with the TGDC this year. Can you begin by explaining 
why you selected this teacher? 

a. Probe: Why do you think he/she would report that it’s a positive process? 

b. Follow-up: How many formal and informal observations have you done with this 
person? 

2. Can you give me a little bit of background about this teacher? 

a. Probe: Can you tell me a little about how this teacher fits into the school 
community? 

b. Probe: In what kinds of ways is the teacher involved in school activities outside of 
his or her classroom (e.g., band, cheerleading)? 

c. How is the teacher viewed by his or her colleagues? 

3. Thinking back to the beginning of this school year (2013-2014), how would you 
describe your relationship with this teacher? 

a. Probes: Now, after the cycles are complete, what do you think about your 
relationship with this teacher? 

b. What are some of the strengths of your relationship with this teacher? 

c. What about areas of growth? 

4. What were your overall goals working with this teacher this year? 

a. Follow-up: Why do you think these are important and should be the priorities? 

b. Do you think the teacher shares the same goal with you? Why? 

5. We sat in the training with you and we know you were told to use the Teaching and 
Learning Framework and the TGDC process in certain ways. We also understand, 
however, that people use this tool in different ways. With XXXX teacher, could you 
describe how you used the preobservation conferences? 

a. Probes: How long did the conference(s) last? 

b. Where did it/they take place? 

c. What was the focus of the conference(s), and how did the conversation(s) go? 
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6. Could you describe how the observations went with this teacher? 

a. Probes: How long was the lesson(s) you observed (e.g., whole lesson, part of a 
lesson)? 

b. Did the teacher follow the lesson plan(s)? 

c. Did the preconference activities seem to matter to the lesson(s)? If yes, how? 

7. Could you describe how the postobservation meetings went? 

a. Probes: How long did the meetings last? 

b. Where did it/they take place? 

c. What was the focus of the conference, and how did the conversation go? 

d. What feedback did you give to the teacher about the lesson, and what was the 
teacher’s reaction? 

8. If we could speak to the teacher and ask him or her what (if anything) he or she 
learned from the TGDC this year, what do you think the teacher would say? 


Teacher 2 

1. Now tell me about the second teacher you selected — the person whom you thought 

would report a negative experience with the TGDC this year. Can you begin by 

explaining why you selected this teacher? 

a. Probe: Why do you think he or she would report that it’s a negative process? 

b. Follow-up: How many formal and informal observations have you done with this 
person? 

2. Can you give me a little bit of background about this teacher? 

a. Probes: Can you tell me a little about how this teacher fits into the school 
community? 

b. Probe: In what kinds of ways is the teacher involved in school activities outside of 
his or her classroom (e.g., band, cheerleading)? 

c. How is the teacher viewed by his or her colleagues? 
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3. Thinking back to the beginning of this school year (2013-2014), how would you 
describe your relationship with this teacher? 

a. Probes: Now, after the cycles are complete, what do you think about your 
relationship with this teacher? 

b. What are some of the strengths of your relationship with this teacher? What about 
areas of growth? 

4. What were your overall goals working with this teacher this year? 

a. Follow-up: Why do you think these are important and should be the priorities? 

Do you think the teacher shares the same goal with you? Why? 

5. You’ve walked me through how you have been using the pre, obs, and post with the 
first teacher. Have you used the pre, obs, and post with the second teacher in similar 
ways? How has each of these gone? 

6. We sat in the training with you and we know you were told to use the Teaching and 
Learning Framework and the TGDC process in certain ways. We also understand, 
however, that people use this tool in different ways. With XXXX teacher, could you 
describe how you used the preobservation conferences? 

a. Probes: How long did the conference(s) last? 

b. Where did it/they take place? 

c. What was the focus of the conference(s), and how did the conversation(s) go? 

7. Could you describe how the observations went with this teacher? 

a. Probes: How long was the lesson(s) you observed (e.g., whole lesson, part of a 
lesson)? 

b. Did the teacher follow the lesson plan(s)? 

c. Did the preconference activities seem to matter to the lesson(s)? If yes, how? 

8. Could you describe how the postobservation meetings went? 

a. Probes: How long did the meetings last? 

b. Where did it/they take place? 

9. What was the focus of the conference, and how did the conversation go? 
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10. What feedback did you give to the teacher about the lesson, and what was the 
teacher’s reaction? 

11. If we could speak to the teacher and ask him or her what (if anything) he or she 
learned from the TGDC this year, what do you think the teacher would say? 

End-of-Year Interview 

Focus Administrator: Year 2 End-of-Year Interview 

Thank you for agreeing to help us understand how administrators learn to use observation 
protocols in teacher evaluation systems. In the last interview we conducted with you, we asked 
you to reflect on the TGDC process with just two teachers. But for this interview, we will be 
asking you to reflect more globally on the process as a whole this year with all of your teachers. 
The whole session should last no more than 60 minutes. If you need to take a break at any point, 
just let me know. 

Like with previous interviews, we will keep everything you say confidential, and the 
results from this work will not be used in any way to evaluate you or anyone participating in the 
research. If you agree, I will audio record today’s interview just so that I can refer back to it 
when completing my notes. We will keep the audio recordings until the data cleaning and 
analysis are complete. The recordings will then be destroyed. If this is all OK with you, I will go 
ahead and turn on the recording in just a moment. 

1. We realize that TGDC sort of dropped in from the sky. But you were busy leading 
your staff, doing teacher evaluation, and working on the improvement of teaching and 
learning before TGDC showed up. We would like to understand who you were as a 
leader before TGDC. If I could have been in your school prior to TGDC, how would 
your teachers have described you as a leader? Can you describe your leadership style 
to me? 

2. How does the TGDC fit with your leadership approach? 

3. Now that you’ve been working with the TGDC for the last 2 years, what, if anything, 
have you learned from doing all of this? 

4. At this point, what do you think are the strengths and weaknesses of the TGDC? 
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5. With the large number of focus elements, which of them do you focus on with your 
teachers? 

a. Probe: Are there particular elements that you find are most critical for effective 
teaching? 

6. How do you approach/work with teachers who are reluctant to change their practice? 

7. What kinds of things have you heard your teachers say to each other about the 
TGDC? 

8. What parts of the TGDC process do you feel most confident in doing? Which aspects 
of the process, if any, would you identify as still “developing” or where you would 
like to further improve? 

9. What do you do differently now in using TGDC from what you did a year ago? 

10. There are at least two goals to the EGDC observations: evaluating teachers and 
helping teachers improve. How do you juggle those two goals when you are 
conducting observations? 

11. How did you go about assigning the final evaluation scores for your teachers this 
year? 

12. How did those scores fit with the work you did on TGDC? 

13. Given how diverse LA is as a district, we want to make sure we’re attending to 
issues of race and class in the interactions between administrators and their teachers. 
Are there ways in which race and class come up in this context? 

a. If so, how? 

Support for the interviewer: For example, sometimes these issues show up in little 
ways — such as when a White teacher has low expectations for children of color. 
Sometimes they show up in big ways, such as when a principal of color is 
perceived as being extra hard on a White teacher in the evaluation. 

14. Finally, as you look ahead to the TGDC cycle next year, what are your plans? 

a. Follow-up: How will you/have you selected teachers to participate in the TGDC 
next year? 
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Think-Aloud: Focus Administrators 
Year 2 UCAST Think-Aloud Protocol 

Directions for interviewer. Please read through all of the directions (the ones below and 
the ones you will read to the participant) prior to the think-aloud. Text in bold you should read to 
the participant so that our approach is standardized across interviewers. 

Focus Administrator Think-Aloud 

Thank you for agreeing to help us understand how observers learn to score. Today, we 
are going to try to understand how you think about Domains 2 and 3 of the teaching and learning 
framework. In order to do that, we are going to watch a video together with a transcript in front 
of us and then listen to you as you assign scores to what you saw. 

Some of what we do today will be similar to what you do in your usual observations, and 
some of it will be different. I will describe the whole process to you so you have a big-picture 
view, and then I’ll answer any questions you have. First, we will watch the short video together. 
You can follow along on the transcript, if you would l ik e. You can take additional notes, but you 
do not have to. After the video is over, I will ask you to align the evidence (by cutting and 
pasting from the transcript to the scoring sheet) and then give a score to just three of the focus 
elements. While you do all of that, I will be quiet, watch, and listen. I won’t interrupt you, but I 
will write down places where I don’t understand something you have said. I will wait and ask 
you my questions at the very end. 

I mentioned that we call this a think-aloud protocol. Think-aloud protocols try to 
understand how someone does something by listening to the person talk out loud as he or she 
does the thing. So in the parts where I ask you to speak out loud, you just need to say the 
thoughts that are going through your mind. When we have done these think-alouds with other 
principals, some have said they like it, others have said it feels strange. Whatever your reaction, 
please do not worry about what you are saying. There are no silly things to say. We know so 
little about how administrators do the work of observing, so it is really helpful just to hear people 
think. 

Would it help if I give you some examples of the kinds of things you might say? [Pause 
for response] 
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When you are aligning evidence, you might say something like, “I am trying to find the 
spot where there is evidence of the teacher’s questioning” or “I’m trying to decide if I want to 
use this quotation as evidence for questioning and discussion or for using assessment.” When 
you are assigning scores, you might say something like, “I don’t think the teacher did a very 
good job asking questions. They were mostly low level.” 

After you are done scoring, I will ask you some questions about what you said. And then 
we’ll be all done. I think this should take us about an hour. 

Does all that make sense? [Pause for response] 

Do you have any questions before we get started? [Pause for response] 

Before we begin, let me give you a little bit about the background of this video. This is a 
first-grade “regular” classroom. The video starts at the beginning of a lesson. The video shows 
about 13 minutes of math instruction and activities, and the students have had a little bit of 
instruction on odd and even numbers, but not a lot. So the content is relatively new. 

[Begin video. Video is 13 minutes long.] 

After video is over. OK, the next step is for you to align and rate as you normally would. 
You will use the electronic copy of the transcript to cut and paste at least one piece of evidence 
for each of the three elements. If you would like to include more than one piece of evidence, you 
can, but you don’t have to. When you decide on your overall rating, you should use all the 
evidence, but you do not need to include all the evidence on the rating sheet. Again, I will be 
quiet, watch, and take notes as you talk out loud. OK? Questions? 

Before you start, sometimes it is hard for me to get everything down; would it be OK if I 
record you so I can get complete notes? I will delete the recording once I have completed my 
notes. 

The interviewer will have the administrator proceed though the steps above. The 
interviewer should focus on understanding (a) the rationale that the administrator uses in any 
aspect of describing evidence, sorting evidence, or assigning and/or justifying ratings and (b) the 
places where the administrator is using information that comes from somewhere other than the 
video to make decisions about ratings. 

So that we can learn a little bit more about how administrators are thinking, we are asking 
everyone some clarifying questions about the elements you scored. So I will start with those: 
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1. For Expectations for Learning & Achievement, you gave the teaching a XX. Can you 
tell me a little bit more about why you gave it a XX and not a XX - 1 or a XX +1? 

a. Probe: How did you decide what level of expectations the teacher had? 

b. Probe: How did you decide how many students the teacher had those expectations 
for? 

2. For Quality/Purpose of Questions, you gave the teaching a XX. Can you tell me a 
little bit more about why you gave it a XX and not a XX - 1 or a XX + 1? 

a. Probe: How did you decide what kinds of questions were being asked? 

b. Probe: How could you tell if the teacher was differentiating the questions for 
students? 

3. For Stcindards-Based Projects, Activities & Assignments, you gave the teaching a XX. 
Can you tell me a little bit more about why you gave it a XX and not a XX - 1 or a 
XX + 1? 

a. Probe: How did you decide whether the activities were rigorous and appropriate 
for the students? 

b. Probe: How did you decide whether students were cognitively engaged by the 
activities? 

4. Ask any other clarifying questions you may have. 

Now I’d like to take a step back and get a little perspective on how this teaching and 
rating made sense to you. 

1. How does what we just did compare with what you have been doing with the teachers 
at your school this year? 

2. Which was the hardest element to rate for this video? Why? 

3. Which was the easiest element to rate for this video? Why? 

4. What’s your overall assessment of this excerpt of teaching? 

a. [If needed] What makes you say so? 

b. At what point did you arrive at that impression in your mind? 

5. Are there any other things I should know about how you think about rating before we 
end today? 
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OK, that is my last question. Could you please save your rating sheet and e-mail me a 
copy? I really appreciate your help. 

Thank you again. I know you are very busy, so thank you very much for making the time 
to do this with me. 
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Glossary 

To assist the reader, we summarize terminology used throughout the report that is 

specific to the Los Angeles Unified School District or to our study. Further description of each of 

the following terms can be found in the body of the report. 

Evaluation Context 

certification. All administrators in the district are required to be certified on the TLF. The TGDC 
training culminates in a certification test in which administrators review a lesson plan, 
watch the videotaped lesson, and review the postobservation conference. 

Education Sendee Center (ESC). LAUSD also has four ESCs (North, South, East, and West) 
through which summer and school-year TGDC trainings are organized. 

Educator Growth and Development Cycle ( EGDC ). The EGDC is the overarching evaluation 

system in LAUSD, with specific versions tailored to evaluate central office staff, school- 
level administrators, and teachers. 

local school district (LSD). LAUSD is made up of eight local school districts, or LSDs, which 
are geographical zones based on family residence. 

My Professional Growth System (MyPGS). MyPGS is the online platform through which 

administrators enter data (e.g., scripted evidence from observations, final observation 
ratings) throughout the TGDC evaluation process. 

School Leader Growth and Development Cycle (SLGDC). For school leaders, including 

principals and assistant principals, the SLGDC process is parallel to the TGDC process, 
consisting of formal observations to assess the quality of administrators’ pre- and 
postobservation conferences with teachers as well as that of the pre- and postobservation 
conferences conducted by district officials. 

STULL. This is the observation-based evaluation system that predated TGDC. In the STULL 

process, areas of evaluation included achievement of instructional objectives, preparation 
and planning, classroom performance, and general professional skills. 

Teacher Growth and Development Cycle (TGDC). The TGDC is the teacher evaluation system to 
which we refer throughout the report. The TGDC consists of formal goal setting at the 
beginning of the year, two formal observations with pre- and postobservation 
conferences, informal observations, and a formal summary evaluation. 
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Teaching and Learning Framework ( TLF ). The TLF is LAUSD’s name for its observation 

protocol, which is a modified version of Danielson’s Framework for Teaching (Danielson 
& McGreal, 2000). 

TGDC training. Training is used to describe the mandatory 4-day training exercise completed by 
all administrators prior to conducting TGDC observations. Most participants in our study 
were trained during the summer, although a small number were trained during the school 
year. 

Personnel 

administrator. An administrator is defined as any school- or district-level leader tasked with 
conducting formal TGDC observations. The administrators in our study are primarily 
school leaders (i.e., principals and assistant principals). 
focus administrator. A focus administrator is one of the 38 volunteer administrators who were 
selected to participate in additional data collection activities for the research team, 
including surveys, interviews, and think-aloud exercises. 
instructional director. Each LSD has an instructional director who coordinates evaluation 
activities and supplemental training for administrators within his or her district. 
teaching and learning coordinator ( TLC ). A teaching and learning coordinator is an individual 
hired to support administrators in their LSDs. 
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Notes 

1 Although alignment is the formal term for matching observation evidence with TLF elements, 

tagging is also used in less formal contexts to refer to the same process. 

2 Although we know that administrators also devoted time to TGDC over the summer, our data 

source (the end-of-year survey) focused on hours in a typical school week. 
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